Claims audit

Each claim below has its verdict, the conditions it holds under and links to the files that show it. For the 2025 ML project, each corrected claim keeps its original wording beside the correction.

Verdicts

Supported
The cited files show it, as stated.
Supported with limits
The cited files show it, but only under the conditions named in its limits. Quote it with them.
Not supported
The cited files do not show it, or show something else.
Artifact
The number is real, but it measures a bug or a setup quirk rather than the thing it was labelled as.

The rk run

The rk run's Q15 results come from epoch 1. They hold inside Q15 fixed point with floor rounding at the run's fixed cycle budget, under the conditions each claim lists. In float64, rk4 is far more accurate than the champion.

Supported: R6, R7, R9, R11, R12, R13, R14. Supported with limits: R1, R2, R3, R4, R5, R8, R10.

R1: The Q15 lead

Supported with limits Headline claim

Claim: Inside Q15 fixed point with floor rounding, at a 65,536-cycle budget, on the analytic m0plus_fast cost model with magnitude weighting, the searched three-stage method 11e898cb has 2.95x lower held-out error than midpoint, the best of eight classical methods.

Limits: This is an epoch-1 result on the cost model that the trace later found had rk4 and rk38 in the wrong order; epoch 2 scores under a corrected model and has not re-measured it. Elites were picked on held-out error among 141,364 archived tableaus, so the four held-out problems act as a selection set and the lead carries winner's-curse bias. The 2.95x is an RMS over four problems: the champion has lower error than midpoint on three of them and higher on pendulum, and much of the gap comes from rc_thermal, where every classical method stalls near a quantization floor. Of six cost-basis and weighting cells, the champion keeps its lead on every leave-one-out subset in two, both analytic: 2.95x under magnitude weighting and 3.36x under reference-norm weighting. On the traced whole-step basis the ratio is 1.01x under magnitude weighting (lowest leave-one-out 0.614x) and 1.84x under reference-norm weighting (lowest 0.604x). Under median-anchor weighting, which weights the four problems equally, it is 1.1x on the analytic basis (lowest 0.814x) and 0.0494x on the traced one; the source reads that last cell as the weighting magnifying pendulum, not as the champion losing by that factor. The lead is a floor-rounding result: under round-to-nearest, at the same budget and cost model and with no extra cycles charged for the rounding, rk38, one of the four classical methods run under both rounding modes, reaches 0.0263 held-out error, below the champion's 0.0286 under floor rounding.

R2: Every discovered cell elite ahead of all eight classical methods

Supported with limits Headline claim

Claim: In all 16 of the 18 occupied archive cells that a discovered method holds, it has lower held-out error than all eight classical methods, whatever their cost; the closest, a four-stage elite at 30 cycles per step, has 0.0836 against midpoint's 0.0844.

Limits: Same conditions and the same selection bias as R1: Q15 with floor rounding, 65,536-cycle budget, analytic cost model, magnitude weighting. Classical methods hold the other 2 cells. Re-ranking all 141,364 archived tableaus on the analytic basis, no classical method ranks above the champion under any of the three weightings, though the champion's own rank moves with the weighting: rank 1 under magnitude weighting, 15 under reference-norm weighting and 9,974 under median-anchor weighting.

R3: Leave-one-out

Supported with limits

Claim: Dropping any one of the four held-out problems leaves the champion ahead, by 1.92x to 3.17x.

Limits: On the published basis only (analytic cost, magnitude weighting). The low end, 1.92x, comes from dropping rc_thermal, the problem where every classical method stalls near a quantization floor. Magnitude weighting lets problems with small errors count for little, so this check cannot show a loss on one of them: the champion has higher error than midpoint on pendulum.

R4: Out-of-sample problems

Supported with limits Headline claim

Claim: On eight out-of-sample problems, the best of the three discovered methods tried had lower Q15 error than the best of the five classical methods run (euler, heun2, midpoint, rk4, rk38) on 4 of 5 non-stiff problems. On the stiff robertson_scaled problem every discovered method overflowed, and so did rk4 and rk38; euler, heun2 and midpoint finished, midpoint with the lowest error.

Limits: Out-of-sample, not blind: no optimizer or model saw these problems and the champion was fixed before they ran, but people chose the problems after the search began. The fixed champion alone had the lower error on 3 of the 5 non-stiff problems.

R5: In float64, rk4 is far more accurate

Supported with limits Headline claim

Claim: In float64, rk4 is far more accurate than the champion: at the step count the 65,536-cycle budget gives each method, the champion's float64 error was 1,992x to 420,975x rk4's on the 7 validation problems where both finish.

Limits: The champion is order 2 and rk4 order 4, and classical order-2 methods trail float64 rk4 by a similar margin in the same runs. Two wider comparisons, run on the seven search and held-out problems rather than the validation set, measure the arithmetic rather than the coefficients: float64 rk4 had lower error than every Q15 run, classical and discovered alike, in 56 of 56 fixed-step cells at identical step counts, and at tolerances of one Q15 least significant bit the most accurate library solver, Radau on each of those problems, was a median 519x more accurate than the most accurate Q15 run. Dormand-Prince (SciPy RK45) was among the library solvers but was never the most accurate one, and it was never run against the champion in the same arithmetic. The search optimized for Q15 floor arithmetic on purpose.

R6: The champion appeared at cycle 33

Supported

Claim: The champion appeared at cycle 33, and no later cycle in epoch 1's 3,931 lowered the best held-out error.

Limits: The search kept filling and improving other cells after cycle 33: 11 of the 16 discovered cell elites appeared later, including elites of order 3 and order 4; the last new cell was filled on 2026-09-05 and the last cell improvement came on 2026-09-09.

R7: An audit caught the cost model's error

Supported Headline claim

Claim: A host-side audit compiled the step with GCC 13.2.1 and traced it under an emulator. The compiled step agreed with the Python evaluator bit for bit in 144 of 144 cases, 25 of them overflow cases where both stop at the same operation, and it showed that the m0plus_fast cost model the archive was scored under put rk4 and rk38 in the wrong order. The run froze epoch 1 and started epoch 2 under a corrected cost model instead of rescoring the old archive.

Limits: The emulator is instruction-accurate, not cycle-accurate. Cycle counts come from a timing table applied to the executed instructions; nothing was measured on a physical chip.

R8: Floor rounding reorders the classical methods

Supported with limits

Claim: Floor rounding reorders the classical methods: on the search problems euler has the lowest error and rk4 the highest, and round-to-nearest puts rk4 back ahead of euler.

Limits: The round-to-nearest comparison covers four methods (euler, heun2, rk4, rk38). Euler's lead holds on the search set under the published basis only; on the held-out problems rk38 has the lowest error of those four under both rounding modes.

R9: The premise test came back mixed

Supported

Claim: The project's own premise test, with thresholds committed before the search, came back mixed: rk4 on both cost models and heun2 on the fast one met the proceed criterion, while heun2 on the slow-multiplier model met neither the proceed nor the kill criterion.

Limits: The thresholds were committed to the project's own repository before the test ran, not to an outside registry. A later comparison of four classical methods on the seven search and held-out problems, made after the search began, found rk4 best on 0 of 7 under floor rounding and 1 of 7 under round-to-nearest.

R10: Measured speed and the cost model

Supported with limits

Claim: Measured in Python on the same solver path, the champion's step ran 1.45x faster than rk4's against a model prediction of 1.5x, and analytic cycles tracked measured time with Pearson r = 0.97 over 56 runs.

Limits: Python wall clock, so it speaks to ordering, not to absolute cycle counts on a Cortex-M0+, and only as far as Python mirrors the chip: Python timed rk4 slightly faster than rk38, as the analytic model predicts, while the compiled trace found rk38 cheaper on a Cortex-M0+ (R7).

R11: A scorer the search cannot edit

Supported Headline claim

Claim: The scorer is out of the search's reach: the harness is mounted read-only, and a sha256 over the pinned files (10 in epoch 1, 14 since epoch 2) is checked at every start and stored in every record. 91 golden and canary cases must pass before any cycle runs, and the suite collects 1,908 tests.

Limits: No GitHub credential enters the container: it commits and the host pushes. The one credential inside is the model's sign-in file, mounted read-only. The counts are as of 2026-09-23.

Evidence

R12: How the run ran

Supported

Claim: Epoch 1 ran from 2026-08-29 to 2026-09-10: 3,931 search cycles over 13 days of archive and 141,364 scored records, all under the epoch-1 hash apart from 21 written minutes before that pin was set. No person chose candidates or scores and the pinned scorer never changed, though the runner started 24 times, including restarts to deploy changes to the unpinned harness code. Epoch 2 started on 2026-09-17 under the new verifier hash, stopped late that day, and nothing restarted it until 2026-09-23.

Limits: Autonomous means no person chooses what the search tries or how it scores; it does not mean free of human operation. The watchdog now resumes the stops it makes itself; after a reboot the run is started by hand with one command.

R13: A language model steers the search inside a narrow schema

Supported

Claim: A language model proposes each search cycle's direction as a JSON directive that can only narrow the search, and an unknown key sends the runner to a fixed fallback. The scorer, the verifier and every hypothesis verdict are code.

Limits: The model is shown held-out errors (U2), so those errors are not independent of the search.

R14: An independent reimplementation agrees

Supported

Claim: The overview site's browser demo reimplements the Q15 solver in JavaScript and reproduces the Python evaluator in 154 of 154 fixture cases.

Evidence

The 2025 ML project

The project's benchmark ran, and its evaluation harness works for explicit tables, both with limits. Its main conclusions from 2025 did not hold.

Supported: N10, N11. Supported with limits: N1, N9. Not supported: N2, N3, N4, N7, N8. Artifact: N5, N6.

N1: The 2025 benchmark ran

Supported with limits Headline claim

Claim: Across 12 methods and 10,000 generated ODEs (8,113 scorable), Dormand-Prince had the lowest mean error (0.0791) and RK4 the next lowest (0.14).

Limits: The error is the mean of each ODE's maximum error, which a few outlier ODEs dominate. The trial entries were copies of RK4 and Dormand-Prince or random tables, the Gauss-Legendre entries ran through a stepper that ignores the implicit part of a table (N5), and the reference solutions came from SciPy RK45, an adaptive Dormand-Prince solver, so the Dormand-Prince entry is measured against a reference built on its own tableau. This ranks two textbook methods against copies of themselves, random tables and truncated methods.

Evidence

N2: The rediscovery claim (not supported)

Not supported Headline claim

Claim: Evolutionary search independently rediscovered RK4 and Dormand-Prince.

Original wording:

Evolutionary search, by contrast, independently rediscovered RK4 and Dormand-Prince with exact performance match (README).

Correction: RK4 was placed in the starting population of the 4-stage runs and Dormand-Prince in that of the 7-stage run. The genetic operators did run and the population moved off the seed, but the composite score is clipped at 1: every one of the 10,000 rows of the one training log scores 1, and the checkpoints of trials 12 to 15 record a best and a mean score of 1 in every one of their 100 epochs. The best-table tracker replaces the saved table only on a strictly higher score, so the seed stayed. Trials 12, 14 and 15 saved RK4 exactly and trial 13 saved Dormand-Prince exactly.

Evidence

N3: The optimality claim (not supported)

Not supported

Claim: The results validate RK4 and Dormand-Prince as mathematically optimal.

Original wording:

validating them as true mathematical optima (README); RK4 and Dormand-Prince are not just historically successful methods, but mathematically optimal solutions (results/analysis_report.md).

Correction: Optimality would need a stated objective and a proof or a search that could have found something better. Here the score could not tell RK4 from a random table: trial 8's saved file gives its seeded random table a score ratio of 1 to RK4 at 69.4 times RK4's error on the same sample.

Evidence

N4: The local-minima claim (not supported)

Not supported Headline claim

Claim: Gradient descent collapsed into local minima.

Original wording:

Empirical benchmarks show gradient descent consistently collapsed into local minima, producing near-identical low-accuracy schemes (README).

Correction: The generator that proposes tables was never trained: its optimizer is created and never stepped, its outputs never pass validation, and every empty slot is filled by a random generator that resets the seed to 42. The only gradient descent in the project trained a surrogate model, and nothing in the training loop reads its predictions. Trials 1-4, 6, 8 and 9 saved the same A matrix and weights b (trials 8 and 9 carry a different node vector c, which none of the 6 test families depends on). Trials 8 and 9 score the same because they are the same table.

Evidence

N5: Gauss-Legendre 3's error (an artifact)

Artifact

Claim: Gauss-Legendre 3 is numerically unstable, with a mean error of 8.31e+211.

Original wording:

Gauss-Legendre 3: Numerical instability (8.31e+211 max error) (results/analysis_report.md).

Correction: The stepper reads the diagonal only while that stage is still zero and never reads the upper triangle, so an implicit table runs as its strictly lower part. Both Gauss-Legendre entries ran as truncated explicit methods of order one; the number measures that bug, not the method.

Evidence

N6: The success rate (an artifact)

Artifact

Claim: Every method succeeded on 81.1% of the ODEs.

Original wording:

Success Rate 81.13% for every method (results/analysis_report.md).

Correction: The rate is a property of the ODE set, not of the methods: the reference solver supports 5 of the 6 families in the final test (Robertson is missing), and some polynomial ODEs blow up before the end of the interval. It is the same number for every method for that reason.

Evidence

N7: The price of forcing variety (not supported)

Not supported

Claim: Forcing novelty produced a different method at a large accuracy cost.

Original wording:

Novelty reward successfully created different method but at severe accuracy cost (results/analysis_report.md).

Correction: Trial 16's table is the top scorer among fresh random draws: the script's evolution branch never ran, so no mutation or crossover happened. Its fitness read an accuracy field that does not exist, so accuracy carried no weight at all. Its mean error of 5,577 describes a random table picked without regard to accuracy, not a price paid for variety.

Evidence

N8: Training scale (not supported)

Not supported

Claim: Large-scale training on 10,000 ODEs.

Original wording:

Large-scale training on 10k ODEs confirmed the framework could replicate classical integrators (README).

Correction: The 10,000-ODE set was used only for the final evaluation. Trials 8 to 16 trained on 1,000 ODEs each and trials 1 to 7 on 500 to 1,200, so no training set came near 10,000. No committed result ran on a GPU, and the author had already removed the CUDA-speedup claim from the README.

Evidence

N9: The 2025 harness

Supported with limits

Claim: The project built a working evaluation harness for explicit tables: a generic Butcher-table stepper, a generator covering 11 ODE families including stiff ones, a SciPy reference solver, parallel scoring and 16 trial configurations, and it ran a 12-method benchmark over 10,000 ODEs. Its source tree holds 10,279 lines of Python across 47 files.

Limits: The harness runs end to end for explicit tables. Two of its parts produced the artifacts in N5 and N6: the stepper runs only the explicit part of a table, and the reference solver covers 5 of the 6 test families and 5 of the 11 training families. An SQLite results module is in the tree, but nothing in the pipeline calls it. Its main conclusions from 2025 are what did not hold.

Evidence

N11: No test suite

Supported

Claim: The repository has no automated test suite: tests/ holds two example scripts, and src/evaluation/quick_test.py is a smoke script.

Evidence

Across both projects

Each of these draws on both projects and names the claims above that it rests on.

Supported with limits: U1. Not supported: U2, U3.

U1: Which methods lead depends on the arithmetic and the cost model

Supported with limits Headline claim

Claim: In float64 the established methods were more accurate in both projects: in the rk run float64 rk4 was 1,992x to 420,975x more accurate than the champion at the same cycle budget, and in the 2025 benchmark Dormand-Prince and RK4 had the lowest mean error.

Limits: The 2025 benchmark is weak evidence: its other entries were copies of RK4 and Dormand-Prince, random tables and truncated methods, and its reference solutions came from SciPy RK45, itself a Dormand-Prince solver (N1). In the rk run Dormand-Prince was never run against the champion in the same arithmetic (R5). Inside Q15 with floor rounding at a 65,536-cycle budget, on the analytic cost model, the searched method leads the classical fixed-step methods (R1); on the traced whole-step basis that lead falls to 1.01x, which dropping one held-out problem overturns.

U2: A blind comparison (not supported)

Not supported

Claim: The comparisons were blind.

Correction: Neither project ran a blind test. In the rk run the model that proposes directions is shown held-out errors and elites are picked on them, so the held-out comparison is not blind; the nearest thing to an independent check is the out-of-sample validation suite (R4). The 2025 project's evolutionary search started from the very tables it was said to rediscover (N2).

Evidence

U3: Large local minima (not supported)

Not supported

Claim: The search landscape is full of large local minima.

Correction: Neither project measured the landscape. The 2025 flat results came from a clipped score, an untrained generator and a random fallback reseeded to 42 (N2, N4). In the rk run the best held-out error stopped improving after cycle 33 while other cells kept improving (R6), which describes one search, not the landscape.