The 2025 ML project
Novel-Numerical-Integration-Methods was a 2025 attempt to find new explicit Runge-Kutta tables with machine learning: a neural generator, a surrogate model and an evolutionary search, scored on generated ODEs. It built an evaluation harness that runs end to end for explicit tables, and its benchmark of 12 methods on 10,000 ODEs ran to completion. Its main conclusions did not hold. The evolutionary search saved the textbook tables it was seeded with, because a clipped score left every candidate tied with the seed, and the neural generator was never trained.
What it set out to do
The plan was to generate Butcher tables for explicit Runge-Kutta methods with a neural network, refine them with a surrogate model and an evolutionary search, and score them on generated ordinary differential equations against four textbook baselines: RK4, Dormand-Prince and Gauss-Legendre 2 and 3. Trials weighted accuracy, efficiency and stability differently, and some added a reward for tables unlike RK4.
The repository has 22 commits. The earliest is dated 2025-08-17, the work itself ran from 2025-09-13 to 2025-09-21, and the last commit, on 2025-12-02, touches only comments and docstrings.
What it built
The evaluation harness runs end to end for explicit tables. It has a generic Butcher-table stepper for explicit methods, a SciPy reference solver, parallel scoring, JSON checkpoints and a training generator with 11 ODE families, including stiff ones such as Robertson, Oregonator and Van der Pol. The reference solver covers 5 of those 11 families, Van der Pol among them but not Robertson or Oregonator. An SQLite results module is in the tree, but nothing in the pipeline calls it.
The final test set came from a separate generator with 6 families: linear_system, van_der_pol, lotka_volterra, brusselator, polynomial and robertson. The reference solver has a branch for 5 of them; robertson is the one it lacks.
The project set up 16 trial configurations, and each saved table records the settings its trial ran with. The whole src/ tree, which holds the generator, the training scripts and the analysis code as well as the harness, has 10,279 lines of Python in 47 files.
The benchmark
The final evaluation ran 12 methods, the four baselines and eight trial tables, over 10,000 generated ODEs in float64, with a fixed step h = 0.01 on t in [0, 1]. Each ODE's reference solution came from Dormand-Prince (SciPy RK45), an adaptive solver. Of the 10,000 ODEs, 8,113 could be scored, the same 8,113 for every method.
For each scorable ODE the benchmark takes the largest absolute error against the reference, then averages those maxima. The lowest mean error, 0.0791, belongs to the Dormand-Prince table and to trial 13, which saved that same table. The next, 0.14, belongs to RK4 and to trials 12, 14 and 15, which saved RK4.
The ranking comes with limits. A few outlier ODEs dominate a mean of per-ODE maxima. The trial entries were copies of RK4 and Dormand-Prince or random tables, and the Gauss-Legendre entries ran through a stepper that ignores the implicit part of a table. Because the reference solutions came from an adaptive Dormand-Prince solver, the Dormand-Prince entry is measured against a reference built on its own tableau. In effect the benchmark ranks two textbook methods against copies of themselves, random tables and truncated methods.
results/real_trial_results.csv at d89c2c809ee0.Data table
| Entry | Kind | Stages | Mean of per-ODE max error | Runtime (s) | Success rate | Note |
|---|---|---|---|---|---|---|
| RK4 | baseline | 4 | 0.14 | 155 | 81.1% | |
| Dormand-Prince (RK45) | baseline | 7 | 0.0791 | 403 | 81.1% | |
| Gauss-Legendre 2 | baseline | 2 | 0.33 | 70.4 | 81.1% | ran as a truncated explicit method |
| Gauss-Legendre 3 | baseline | 3 | 8.31e+211 | 111 | 81.1% | artifact of a stepper bug, drawn off scale |
| Trial 8 | trial | 4 | 0.377 | 153 | 81.1% | same A and b as trials 1 to 4, 6 and 9 |
| Trial 9 | trial | 4 | 0.377 | 151 | 81.1% | same A and b as trials 1 to 4, 6 and 8 |
| Trial 10 | trial | 7 | 0.374 | 367 | 81.1% | |
| Trial 12 | trial | 4 | 0.14 | 146 | 81.1% | same table as RK4 |
| Trial 13 | trial | 7 | 0.0791 | 364 | 81.1% | same table as Dormand-Prince |
| Trial 14 | trial | 4 | 0.14 | 146 | 81.1% | same table as RK4 |
| Trial 15 | trial | 4 | 0.14 | 145 | 81.1% | same table as RK4 |
| Trial 16 | trial | 4 | 5,577 | 150 | 81.1% | the top scorer among random draws |
Runtime is wall time for all 10,000 ODEs with the reference solves included, so it measures more than the stepper. RK4 and trials 12, 14 and 15 ran the same table, yet their runtimes span 145 to 155 seconds, so differences of that size are timing noise. Gauss-Legendre 2 had the shortest runtime, 70.4 seconds, because it ran as a truncated explicit method with 2 stages.
results/real_trial_results.csv at d89c2c809ee0.Data table
| Entry | Stages | Runtime (s) |
|---|---|---|
| RK4 | 4 | 155 |
| Dormand-Prince (RK45) | 7 | 403 |
| Gauss-Legendre 2 | 2 | 70.4 |
| Gauss-Legendre 3 | 3 | 111 |
| Trial 8 | 4 | 153 |
| Trial 9 | 4 | 151 |
| Trial 10 | 7 | 367 |
| Trial 12 | 4 | 146 |
| Trial 13 | 7 | 364 |
| Trial 14 | 4 | 146 |
| Trial 15 | 4 | 145 |
| Trial 16 | 4 | 150 |
What did not hold
The code and the committed results contradict the main conclusions in the README and in results/analysis_report.md. The saved table never moved off the seed the evolutionary search started from, the generator that proposed tables was never trained, and nothing in the project could show that a table is the best possible. Two reported numbers, the Gauss-Legendre 3 error and the shared success rate, describe the harness rather than the methods. Trial 16's large error came from a random table, not from forcing variety, and no training set came near 10,000 ODEs. The project also had no working order check and no automated test suite.
The saved table never left the seed
The project claimed:
Evolutionary search, by contrast, independently rediscovered RK4 and Dormand-Prince with exact performance match (README).
RK4 was placed in the starting population of the 4-stage runs and Dormand-Prince in that of the 7-stage run. The genetic operators did run and the population moved off the seed, but the composite score is clipped at 1: every one of the 10,000 rows of the one training log scores 1, and the checkpoints of trials 12 to 15 record a best and a mean score of 1 in every one of their 100 epochs. The best-table tracker replaces the saved table only on a strictly higher score, so the seed stayed. Trials 12, 14 and 15 saved RK4 exactly and trial 13 saved Dormand-Prince exactly.
Those checkpoints are in trials/trial_01{2,3,4,5}_*/checkpoints/checkpoint_epoch_100.json at d89c2c809ee0. Matching tables the search was seeded with is not a blind result, and neither project ran a blind test.
The score could not tell RK4 from a random table
The project claimed:
validating them as true mathematical optima (README); RK4 and Dormand-Prince are not just historically successful methods, but mathematically optimal solutions (results/analysis_report.md).
Optimality would need a stated objective and a proof or a search that could have found something better. Here the score could not tell RK4 from a random table: trial 8's saved file gives its seeded random table a score ratio of 1 to RK4 at 69.4 times RK4's error on the same sample.
Each trial's saved file compares its table with RK4 on one sample, and trial 8 is not alone. Trials 9 and 10 also score 1 against RK4, at 34.2 and 2,039 times RK4's error, and trial 13, which saved Dormand-Prince, scores 1 at 0.172 times RK4's error. Tables with far higher error than RK4, and one with lower error, all scored the same as RK4.
Every candidate in the training log has the same composite score as well.
The generator was never trained
The project claimed:
Empirical benchmarks show gradient descent consistently collapsed into local minima, producing near-identical low-accuracy schemes (README).
The generator that proposes tables was never trained: its optimizer is created and never stepped, its outputs never pass validation, and every empty slot is filled by a random generator that resets the seed to 42. The only gradient descent in the project trained a surrogate model, and nothing in the training loop reads its predictions. Trials 1-4, 6, 8 and 9 saved the same A matrix and weights b (trials 8 and 9 carry a different node vector c, which none of the 6 test families depends on). Trials 8 and 9 score the same because they are the same table.
The surrogate did train. The checkpoints of trials 12 to 15 show it: each records a surrogate loss in 98 of its 100 epochs (trials/trial_01{2,3,4,5}_*/checkpoints/checkpoint_epoch_100.json at d89c2c809ee0).
| Trial | Stages | Training ODEs | Largest difference from RK4 | Largest difference from Dormand-Prince | Same A and b as |
|---|---|---|---|---|---|
| 1 | 4 | 500 | 1.69 | n/a | trials 2 to 4, 6, 8 and 9 |
| 2 | 4 | 1,000 | 1.69 | n/a | trials 1, 3, 4, 6, 8 and 9 |
| 3 | 4 | 1,200 | 1.69 | n/a | trials 1, 2, 4, 6, 8 and 9 |
| 4 | 4 | 1,200 | 1.69 | n/a | trials 1 to 3, 6, 8 and 9 |
| 5 | 6 | 800 | n/a | n/a | trial 7 |
| 6 | 4 | 600 | 1.69 | n/a | trials 1 to 4, 8 and 9 |
| 7 | 6 | 700 | n/a | n/a | trial 5 |
| 8 | 4 | 1,000 | 1.69 | n/a | trials 1 to 4, 6 and 9 |
| 9 | 4 | 1,000 | 1.69 | n/a | trials 1 to 4, 6 and 8 |
| 10 | 7 | 1,000 | n/a | 12.3 | none |
| 11 | no table saved; not in the final evaluation | ||||
| 12 | 4 | 1,000 | 0 | n/a | RK4; trials 14 and 15 |
| 13 | 7 | 1,000 | n/a | 0 | Dormand-Prince |
| 14 | 4 | 1,000 | 0 | n/a | RK4; trials 12 and 15 |
| 15 | 4 | 1,000 | 0 | n/a | RK4; trials 12 and 14 |
| 16 | 4 | 1,000 | 1.51 | n/a | none |
The Gauss-Legendre 3 error measures a stepper bug
The project reported:
Gauss-Legendre 3: Numerical instability (8.31e+211 max error) (results/analysis_report.md).
The stepper reads the diagonal only while that stage is still zero and never reads the upper triangle, so an implicit table runs as its strictly lower part. Both Gauss-Legendre entries ran as truncated explicit methods of order one; the number measures that bug, not the method.
The success rate belongs to the ODE set
The project reported:
Success Rate 81.13% for every method (results/analysis_report.md).
The rate is a property of the ODE set, not of the methods: the reference solver supports 5 of the 6 families in the final test (Robertson is missing), and some polynomial ODEs blow up before the end of the interval. It is the same number for every method for that reason.
Every benchmark row records the same success rate, 81.13%: 8,113 of 10,000 ODEs.
Trial 16 is the top scorer among random draws
The project claimed:
Novelty reward successfully created different method but at severe accuracy cost (results/analysis_report.md).
Trial 16's table is the top scorer among fresh random draws: the script's evolution branch never ran, so no mutation or crossover happened. Its fitness read an accuracy field that does not exist, so accuracy carried no weight at all. Its mean error of 5,577 describes a random table picked without regard to accuracy, not a price paid for variety.
Trial 16 used its own training loop, and its score is the only one in the project that is not clipped at 1. Over its 100-epoch history the best score peaked at 1.05 in epoch 33, and the saved table is that epoch's random draw. The trial ran for 40,469 seconds.
No training set came near 10,000 ODEs
The project claimed:
Large-scale training on 10k ODEs confirmed the framework could replicate classical integrators (README).
The 10,000-ODE set was used only for the final evaluation. Trials 8 to 16 trained on 1,000 ODEs each and trials 1 to 7 on 500 to 1,200, so no training set came near 10,000. No committed result ran on a GPU, and the author had already removed the CUDA-speedup claim from the README.
In the final code the training generator has 11 families and the reference solver handles 5 of them, so a training set from that generator held ODEs the pipeline could not score. The final test drew from a separate generator with 6 families.
What the project did not check
The project enforced no order conditions: the only hard validation was that the weights sum to one, and its order-4 test uses a condition that RK4 itself fails, so it labels RK4 as order 3. Trials 12, 14 and 15, which saved RK4, and trial 13, which saved Dormand-Prince, are labelled order 3. Every other saved table, trials 1 to 10 and 16, is labelled order 1.
The repository has no automated test suite: tests/ holds two example scripts, and src/evaluation/quick_test.py is a smoke script.
The training log
metrics_log.csv records one training run: 100 epochs of 100 four-stage candidates, 10,000 rows in all. It has no trial column, but only trial 15's saved settings ask for 100 candidates per epoch, so it is most likely trial 15's log. Every logged composite score is 1, the ceiling the score is clipped to, so the score gave the search nothing to select on.
The lowest max_error of any candidate in the log, 2.28e-07, comes at epoch 1, and no later epoch goes below it.
max_error among the 100 candidates in each epoch of the one run in metrics_log.csv, log scale. Every row has a composite score of 1, the clip ceiling. Source: metrics_log.csv at d89c2c809ee0.Data table
| Epoch | Lowest max_error | Median max_error |
|---|---|---|
| 1 | 2.28e-07 | 0.000293 |
| 2 | 1.51e-05 | 0.000702 |
| 3 | 1.82e-05 | 0.001 |
| 4 | 1.31e-06 | 0.000178 |
| 5 | 3.21e-07 | 0.000102 |
| 6 | 1.58e-05 | 0.000638 |
| 7 | 1.26e-05 | 0.00215 |
| 8 | 8.55e-06 | 0.000312 |
| 9 | 6.19e-06 | 0.000198 |
| 10 | 3.61e-07 | 4.94e-05 |
| 11 | 7.61e-06 | 0.000426 |
| 12 | 2.40e-05 | 0.000634 |
| 13 | 5.28e-06 | 0.000216 |
| 14 | 1.58e-06 | 9.74e-05 |
| 15 | 3.87e-05 | 0.000921 |
| 16 | 1.35e-06 | 5.33e-05 |
| 17 | 8.00e-06 | 0.000695 |
| 18 | 3.85e-05 | 0.000949 |
| 19 | 6.06e-06 | 0.000141 |
| 20 | 7.02e-05 | 0.000708 |
| 21 | 3.69e-06 | 9.44e-05 |
| 22 | 3.04e-05 | 0.000851 |
| 23 | 2.50e-06 | 7.41e-05 |
| 24 | 6.12e-05 | 0.00495 |
| 25 | 2.81e-06 | 0.000164 |
| 26 | 7.90e-05 | 0.00065 |
| 27 | 7.79e-06 | 0.000217 |
| 28 | 0.000105 | 0.00465 |
| 29 | 4.09e-06 | 0.000121 |
| 30 | 0.000375 | 0.00563 |
| 31 | 1.24e-06 | 8.72e-05 |
| 32 | 6.39e-05 | 0.00302 |
| 33 | 2.20e-06 | 0.000307 |
| 34 | 0.000131 | 0.00649 |
| 35 | 8.19e-06 | 0.000229 |
| 36 | 2.58e-05 | 0.000691 |
| 37 | 4.75e-06 | 0.000282 |
| 38 | 6.52e-07 | 3.28e-05 |
| 39 | 8.60e-06 | 0.000155 |
| 40 | 5.70e-06 | 0.000567 |
| 41 | 3.12e-06 | 0.000179 |
| 42 | 0.000146 | 0.00754 |
| 43 | 1.41e-05 | 0.000634 |
| 44 | 8.90e-06 | 0.000359 |
| 45 | 1.43e-06 | 0.000136 |
| 46 | 1.30e-06 | 0.00019 |
| 47 | 1.38e-05 | 0.000715 |
| 48 | 3.45e-05 | 0.000813 |
| 49 | 1.98e-05 | 0.000645 |
| 50 | 0.000484 | 0.00629 |
| 51 | 0.000228 | 0.00654 |
| 52 | 2.97e-06 | 0.000153 |
| 53 | 3.37e-06 | 0.000183 |
| 54 | 0.000352 | 0.00553 |
| 55 | 1.16e-06 | 8.01e-05 |
| 56 | 1.25e-06 | 0.000167 |
| 57 | 5.97e-06 | 0.000491 |
| 58 | 2.96e-06 | 6.47e-05 |
| 59 | 1.80e-06 | 0.000131 |
| 60 | 1.19e-06 | 0.000167 |
| 61 | 1.12e-05 | 0.000659 |
| 62 | 2.91e-05 | 0.000707 |
| 63 | 8.53e-06 | 0.000318 |
| 64 | 3.84e-06 | 8.08e-05 |
| 65 | 7.60e-06 | 9.22e-05 |
| 66 | 2.60e-05 | 0.00115 |
| 67 | 1.72e-06 | 5.55e-05 |
| 68 | 1.28e-05 | 0.00062 |
| 69 | 0.000251 | 0.0122 |
| 70 | 1.31e-05 | 0.000737 |
| 71 | 4.77e-06 | 0.000446 |
| 72 | 3.58e-06 | 0.000163 |
| 73 | 1.31e-05 | 0.000158 |
| 74 | 3.76e-06 | 0.000103 |
| 75 | 6.46e-06 | 0.000438 |
| 76 | 2.66e-06 | 0.000172 |
| 77 | 4.78e-05 | 0.00114 |
| 78 | 5.13e-05 | 0.000759 |
| 79 | 1.45e-06 | 4.71e-05 |
| 80 | 8.96e-06 | 0.000194 |
| 81 | 1.43e-05 | 0.000351 |
| 82 | 7.10e-05 | 0.0139 |
| 83 | 1.57e-06 | 0.000127 |
| 84 | 1.36e-05 | 0.000209 |
| 85 | 2.95e-05 | 0.000759 |
| 86 | 6.21e-06 | 0.000151 |
| 87 | 8.15e-06 | 0.000581 |
| 88 | 3.24e-05 | 0.00127 |
| 89 | 4.26e-05 | 0.000603 |
| 90 | 6.98e-05 | 0.00122 |
| 91 | 1.63e-06 | 0.000188 |
| 92 | 9.86e-07 | 0.000106 |
| 93 | 3.95e-05 | 0.00199 |
| 94 | 4.49e-06 | 0.000347 |
| 95 | 4.98e-06 | 0.000405 |
| 96 | 3.18e-05 | 0.00201 |
| 97 | 6.39e-06 | 0.000322 |
| 98 | 3.21e-06 | 8.20e-05 |
| 99 | 5.40e-05 | 0.00176 |
| 100 | 5.95e-05 | 0.0108 |
What still holds
A search like this needs a stepper that takes any explicit Butcher table, a generator of stiff and non-stiff test problems with a reference solver, per-trial configurations and saved artifacts. The 2025 project built all of them, and they run end to end for explicit tables. Two parts have known defects: the stepper runs only the explicit part of a table, and the reference solver has no branch for robertson. The benchmark ran 12 methods over 10,000 ODEs to completion. The report kept every entry, the weakest included, and stated trial 16's mean error of 5,577 as a loss.
The benchmark ranking holds with its limits: in float64 on these ODEs, the Dormand-Prince table and RK4 had the lowest errors. The rk run also found float64 rk4 far more accurate than its champion. Of the two, the 2025 benchmark is the weaker evidence, because its other entries were copies of those two tables, random tables and truncated methods. The rk run looked where the 2025 project did not, in Q15 fixed point with floor rounding.
The author caught some of this in 2025. Trials 6 and 7 put the coefficients that earlier trials kept producing on a forbidden list, so the author had noticed the repeated table, though not its cause. Commit d148c43 changed competitive performance
to significantly worse accuracy
once the 10,000-ODE results came in, though the same commit strengthened near-optimality
to mathematical optimality
. Commit 8c7cdd7 removed the README's CUDA speedup claim and its outperform
claims.