Methodology

How the run measures, checks and reproduces its numbers.

rk-harness
Object of studyExplicit Runge-Kutta tableaus, orders 1 to 4, stages 2 to 6
State arithmeticQ15: int16 in [-32768, 32767], scale 2-15
TargetARM Cortex-M0+, analytic cycle model, two multiplier variants; AVR model advisory only
Evaluation budget65,536 cycles per problem run
Test problems7 fixed (3 search, 4 held out)
Verification gateNine ordered checks, pure code, never raises
Integrity pinsha256 over ten files, checked at container start

rk-harness searches for explicit Runge-Kutta tableaus that give the lowest integration error in Q15 fixed-point arithmetic at a fixed cycle budget, priced by an ARM Cortex-M0+ cost model. Classical tableaus were derived for exact real arithmetic. On a microcontroller with no floating-point unit, roundoff competes with truncation at practical step sizes, and each coefficient costs cycles that depend on its bit pattern[21].

Scoring is pure code: a language model may steer the search and propose hypotheses, but it never scores, tiers or verifies anything. Results are on the other tabs and the machinery on the overview architecture page. Terms are in the glossary below, and a bracketed number such as [21] cites a file in the rk-harness repository, a section of the frozen specification, docs/HANDOFF.md, or a published paper.

  1. Experimental setup
  2. Measurement
  3. Statistical protocol
  4. Verification and trust
  5. Testing
  6. Reproducibility
  7. Limitations
  8. Cost model
  9. Measurement ledger
  10. Related work
  11. Glossary
  12. References

1. Experimental setup

The object of study is the explicit tableau with symbolic order 1 to 4 and 2 to 6 stages. A entries sit on dyadic lattices no finer than 1/32768, and each coefficient is stored as m / 2s with |m| ≤ 32767 and 0 ≤ s ≤ 20. Where no exact pair exists (1/3 becomes 21845 / 216) the closest one is used and the gap is recorded as coeff_quant_error, a measured property rather than a reason to reject[2].

State arithmetic is Q15: 16-bit signed integers at scale 2-15, multiplied as (a * b) >> 15 with an arithmetic shift that floors toward negative infinity, as the ARM ASRS instruction does, and an out-of-range value becomes a verifier rejection. Because the floor is one-sided, each product loses about half a least-significant bit on average, and at small step sizes that bias dominates the Q15 error; the harness copies the hardware on purpose, since code shipped to this target would floor too. The semantics are on the architecture page and under floor rounding[1][21].

Budget and cost accounting

Methods are always compared at an equal cycle budget, never at equal step size. A tableau costing k cycles per step takes n = B // k steps with B = 65,536 cycles, so a two-stage method takes about twice the steps of a four-stage one. Comparing at equal h would hand the expensive methods free work[5].

Cycle counts are analytic, a pure function of the tableau with no compiler in the loop. m0plus_fast (single-cycle multiplier) and m0plus_slow (32-cycle multiplier) are primary; avr_approx is advisory. Each coefficient application is charged the cheaper of a shift-add chain of its CSD weight and a hardware multiply, which is why bit structure matters under a slow multiplier. The tables are in the cost model section below and the counting rules on the architecture page[7][2].

Test problems and the held-out split

Seven fixed initial-value problems drive every error measurement: three form the search set the optimizer sees, and four are held out to catch overfitting. Peaks and scales come from fixtures/problems.json[3].

ProblemSetDynamicsStatest_endFamilyScaleReference
dahlquistsearchy' = -y, y(0) = 1110linear2-2exp(-t)
damped_oscsearchx'' + 2ζωx' + ω²x = 0, ζ = 0.1, ω = 1240oscillatory2-2closed form
vanderpol_mildsearchx'' = μ(1 - x²)x' - x, μ = 0.5220nonlinear2-4mpmath, 30 digits
pendulumheld outθ'' = -sin θ, θ(0) = 1 rad260nonlinear2-2energy invariant
dc_motorheld outaffine RL motor: R = 2, L = 0.5, Ke = Kt = 0.1, B = J = 0.02, V = 125linear2-3matrix exponential
rc_thermalheld outy' = Ay, 3-compartment linear network34stiff2-2matrix exponential
quaternionheld outq' = ½Ω(ω)q, body rates (0.3, 0.2, 0.5)430geometric2-2norm invariant

rc_thermal is the stiff member, with a stiffness ratio of about 70. Its derivative would exceed Q15 range at the state scale, so it carries an extra factor (DERIV_SCALE = 1/8) that the step size undoes exactly. The held-out set cannot be reached from the search module's import graph, and a canary test checks that[3][4][9][17].

2. Measurement

Error is the Euclidean distance between the final state, converted back to physical units, and the reference at t_end, divided by the problem's peak amplitude. pendulum uses relative energy drift |E - E0| / E0 instead, and quaternion the drift of its norm from 1[3].

Search error and held-out error are each the root mean square of the per-problem errors over their set, computed by evaluator.set_error[5]. An RMS sums squares, so a problem's influence on the aggregate grows with the square of its error and the problems do not carry equal weight: the problem a method does worst on dominates that method's score, and which problem that is differs from method to method. The weighting is deliberate. Every per-problem error is already divided by its own problem's scale, so the squares are summed on one shared scale, and that magnitude weighting is what every score in the archive was computed under. The equal-weighting answer is a different scale: divide each problem by the median error of the classical anchors before the squares are summed. The counterfactual under that scale, and under a reference-norm scale that concentrates weight instead of spreading it, is published on the overview results page. The two scales disagree about the outcome, and that disagreement is itself the result.

The Q15 integrator takes n equal steps of h = t_end / n, and an overflow anywhere aborts the run. The evaluator never raises: a failure becomes an inf entry, a zero overflow margin or a None measured order in the ScoreVector it returns, and its m0plus_fast results are primary[4][5].

A verifier then applies nine checks in a fixed order and returns the earliest failure as a coded rejection. Five symbolic checks run before any simulation: NOT_EXPLICIT, ROW_SUM_INCONSISTENT, DYADIC_IMPOSSIBLE, ORDER_NOT_MET (exact residuals over rooted trees) and COEFF_UNREPRESENTABLE. Then come Q15_OVERFLOW, UNSTABLE, NO_ASYMPTOTIC_WINDOW and NAN_OR_INF. The verifier never raises, never calls a language model, never opens a socket and never writes[6][8].

Measured order

Symbolic order says what a tableau should do; measured order says what the implementation does. The float64 integrator runs dahlquist to t_end = 10 at n = 8 · 2k steps for k = 0 to 11. Slopes between consecutive halvings are kept where the error is finite and above 1e-12, the longest run of slopes whose spread stays within 0.08 is fitted by least squares, and the fitted slope is the measured order. With no run of at least two slopes the verifier rejects the tableau with NO_ASYMPTOTIC_WINDOW[5].

The golden tests reproduce the specification's values, rk4 at 4.0706 over 3 points among them. The study runs in float64 because Q15 roundoff would flatten every method's slope; the problem runs measure the Q15 error separately[5][17].

3. Statistical protocol

Code assigns each verified candidate an evidence tier when it is archived. heldout_verified: it improved on the incumbent elite in both search-set and held-out error, across at least two problem families. search_only: the search error improved and the held-out error did not, which is the signature of overfitting. no_incumbent: the cell was empty. no_improvement: there was an incumbent and neither rule above applied, which includes improving held-out error but not search error. Those last two were one word, unreplicated, before the split, and no archived record carries it now. The tier words appear in no prompt template, and a planted tableau tuned to the search set must land in search_only; canaries check both. The archive behind the cells is on the architecture page[11][13].

The two early phases enumerate their spaces completely, so their entries are marked "exhaustive" with the note "optimal within the enumerated space", CMA-ES entries are marked "search result", and tableaus seeded at startup "seeded classical baseline". The lattices are on the architecture page[10][20].

Hypotheses and verdicts

The model may record predictions only as predicates in a closed grammar over archive statistics, and code writes every verdict. A predicate that names an empty cell is inconclusive, because absence of evidence is not refutation. No verdict is issued below the declared minimum sample count, and an effect size below Cohen's d = 0.2 forces inconclusive whatever the comparison says; otherwise the verdict is supported or refuted. Every model call receives the refuted list. The grammar is on the architecture page[12][21].

Falsification protocol

Before the long run started, its premise was tested on rk4 and heun2 in Q15 on damped_osc, against criteria fixed in advance: coefficient arithmetic must be a material share of per-step cost, and Q15 roundoff must dominate the error somewhere in the practical step range. Code computes the verdict (kill, mixed or proceed), and the validation tab publishes it with its thresholds, whichever way it falls[19].

Practical validation

A practical validation suite sits outside the search: equations from real applications that the optimizer never sees and that no score, tier or archive statistic includes. The archived champions and the classical anchors run each one at the same 65,536-cycle budget, to see whether methods selected on the fixed problems hold up on dynamics they were not selected on[5][21].

Two numeric filters run before any win or loss from that suite is published. A problem is flagged degenerate when its finishers' Q15 errors span less than 5 percent from best to worst, because a field that tight is reporting the problem rather than the method; a flagged problem is left out of every tally with its reason printed. The second criterion, an error within 5 percent of the reference solution's norm, catches a state that decayed to nothing. validation/results.json stores that norm for each problem, on the same scale as the errors, so the pages read it from the document rather than assuming it. A third criterion catches the case that defeats both: when any two finishers agree to within 5 percent of each other while both sit within 5 percent of that norm, the field is not separating methods even though one live method widens the spread. Any error ratio within 2 percent of 1.0 is counted as a tie. A further criterion, an identical peak magnitude equal to the initial condition, was dropped because it flags healthy problems[5][20].

4. Verification and trust

The design assumes the language model will try shortcuts. Only the runner talks to it, every path from its output into the system passes a validator, and a directive can narrow the search but never change the objective, the problems, the cost model or the tier rules[15].

The container mounts rk-harness read-only and exits at start-up unless a sha256 over the ten scoring files matches VERIFIER_HASH and the golden tests pass, and every record stores the hash it was scored under. Model-written code runs only inside the problem quarantine, and the container never receives the GitHub token. The architecture page goes through the checks in order[13][14][16].

5. Testing

The golden fixtures are ground truth computed before the implementation existed (the classical tableaus, rooted-tree counts, the rk4-against-rk38 anchor, the Q15 multiplication vectors and a hand-counted ARM sequence), kept verbatim under fixtures/ so the tests check against an outside truth instead of ratifying the specification. A preflight script runs every machine-checkable item of the review checklist, and the architecture page describes the suite[17][18].

6. Reproducibility

The Q15 core is exact integer arithmetic, the evaluator reads no clock, starts no threads and draws no random numbers, and the search is seeded, so two runs from one seed write byte-identical archives. This site is rebuilt with no clock reads, so the same archive always gives the same pages; the architecture page has the detail[5][9][11][20].

7. Limitations

The cost model is analytic and has never been checked against silicon inside the loop: an assembly fixture and a reference C emitter anchor it, but no hardware measurement feeds a score. Derivative evaluation is left out of the cycle count, which is harmless between methods with the same stage count and a real simplification otherwise[7].

Measured order comes from one scalar problem in float64, so it certifies the coefficients, not fixed-point behavior across the suite. Stability extents are sampled along the real and imaginary axes only. Error is measured at the final time, so a method could trade path accuracy for endpoint accuracy without penalty. Floor rounding is one hardware convention; a target that rounds or saturates would shift every Q15 result[5][1].

The search space is narrow by design, and optimality from the enumeration phases holds only inside the enumerated lattices. Four held-out problems make held-out verification a strong filter, not a statistical guarantee, and the two-family rule and the d = 0.2 threshold are conventions chosen in advance, frozen by the specification so results stay comparable across the run[21].

Cost model

Cycles per integration step, computed from instruction counts. The cycle budget and the cycle buckets on the other tabs are built from these numbers.

Anchor: rk4 and rk38

Analytic cycles per step for rk4 and rk38 under the fast and slow multiplier cost models. Bar height is the per-step cost and the printed number is the exact cycle count. Which of the two anchor methods is cheaper swaps between the multiplier models, and that swap is the cost model's main sanity check.
m0plus_fast (1-cycle multiplier)m0plus_slow (32-cycle multiplier)
rk4 under m0plus_fast: 33 cycles per step33rk4 under m0plus_slow: 85 cycles per step85rk4rk38 under m0plus_fast: 36 cycles per step36rk38 under m0plus_slow: 64 cycles per step64rk38
Every plotted bar, in one table (4 rows)
methodcost modelcycles per step
rk4m0plus_fast33
rk4m0plus_slow85
rk38m0plus_fast36
rk38m0plus_slow64
tableaun_statesm0plus_fastm0plus_slow
rk413385
rk4266170
rk44132340
rk3813664
rk38272128
rk384144256

Classical tableaus (n_states = 1)

tableaustagesm0plus_fastm0plus_slowavr_approx
euler1556
heun22131330
heun33235090
kutta332665106
midpoint2111120
ralston22163062
rk3843664132
rk443385134

avr_approx is an approximate model, shown for context only. It never drives the search or the archive grids. A tableau of simple dyadic values can cost much less than its stage count suggests.

Model parameters

modelmuladdshiftloadstore
m0plus_fast11122
m0plus_slow321122
avr_approx142822

m0plus_fast and m0plus_slow differ only in the multiplier. Every archive cycle count is these five numbers applied to a tableau's instruction sequence, never a hardware measurement. The off-archive lanes add their own stated terms, named on the implicit and adaptive tabs.

Measurement ledger

Every measurement the run takes outside the scored archive is a line in rk-work/sidetrack/ledger.jsonl, stamped with the code hash that produced it and the artifact it was read from. The ledger holds 103 points across 10 jobs. The readings are on the implicit and adaptive tabs. The newest measurement ran under code hash 305c692a1b5c.

The code-hash rule

Every artifact is a pure function of the code and the point's parameters, so measuring a point again reproduces it byte for byte. A point counts as measured only under the code hash that measured it, a digest over the executor and the prototype modules, so editing a prototype re-opens its points instead of leaving stale numbers beside fresh ones.

Points in the ledger were measured under 2 code hashes: 305c692a1b5c085a, a98acb39fb4f31ca.

The retry policy

A point that raises is recorded with its message and retried on later firings; after three failures under one code hash it is set aside and listed here. A side-track failure never costs a cycle, because the executor catches it and the run continues.

Rounding error in time integration is an active line of work, and this project did not discover it. Croci and Giles analyse Runge-Kutta discretizations of the heat equation in low-precision floating point and show that under round-to-nearest the computed solution stagnates once the timestep is small enough, with global rounding error growing like unit roundoff divided by the timestep until it does[22]. Hopkins and colleagues measure stochastic rounding and reduced-precision fixed point in ODE solvers built for neuromorphic hardware, the same arithmetic regime as this work on different hardware[23]. Croci and Rosilho de Souza build mixed-precision explicit stabilized Runge-Kutta methods and show where the low-precision part can sit without costing accuracy[24].

The crossover this run measures, where Q15 error turns back up as the step size falls, is the fixed-point counterpart of that stagnation result in a different arithmetic. Finding the same shape under a different rounding rule is a check on this harness rather than a new phenomenon, and it is consistent with what that literature shows.

Two things here are different from that line of work, and both are narrow. The arithmetic is fixed point with directed rounding: every Q15 multiply ends in an arithmetic right shift that floors, so the error is biased one way, where round-to-nearest is symmetric and stochastic rounding is unbiased by construction. And the object of study is the tableau rather than the arithmetic: this run searches coefficient space against that bias and prices every coefficient on a cost model, where the work above analyses fixed methods under a rounding mode. Neither difference is a claim of priority over that work.

Glossary

Terms used across this site, in alphabetical order. Each definition says what the code does.

anchor methods
rk4 and rk38, two classical four-stage, order-4 tableaus with the same stability polynomial, so they differ in cost only through coefficient arithmetic; the cost model section compares them.
Cohen's d
The effect size on a hypothesis verdict: the difference of two cell populations' means over their pooled standard deviation, with the 0.2 threshold of section 3.
cost bucket (cycle bucket)
A log2-spaced bin of cycles per step under m0plus_fast, one axis of the archive grids: bucket 0 is under 16 cycles, bucket 1 is 16 to 31, and so on up to bucket 7 at 1024 cycles or more.
CSD weight
The fewest nonzero signed power-of-two terms that write an integer multiplier m. The cost model charges a coefficient the cheaper of a shift-add chain of that length and a hardware multiply.
cycle budget
The fixed allowance of every archive evaluation, 65536 cycles per problem per cost model, so archive errors compare methods at equal budget (section 1).
cycles to tolerance
The modeled cycles a method needs to reach a fixed accuracy target, whether by more steps, a tighter tolerance or more Newton work per step. The lanes measure it on the axis-T ladder, one fixed list of final-state error targets run on every validation problem, and never mix it with the archive's fixed cycle budget.
directive
A JSON search instruction from the planning model, checked against a strict schema and stamped on every record it produces. Ids starting with D-E mark exhaustive enumeration cycles.
dyadic rational
A number m / 2^s, which binary fixed-point hardware applies exactly, so A entries are snapped to dyadics and every coefficient is stored as an (m, s) pair. The b weights are solved exactly from the order conditions instead, since snapping them would put order 3 out of reach.
elite
The one record a MAP-Elites cell holds: the verified tableau with the lowest held-out error for that order, stage count and cost bucket, displaced only by a strictly lower one.
embedded pair
Two Runge-Kutta methods sharing one A matrix and one set of stages, with weight vectors of different order whose difference estimates the local error. An archive record holds one b vector, so a pair cannot be scored there and is measured off-archive.
floor rounding (ASRS)
The Q15 multiply rounds toward negative infinity, as the ARM ASRS shift does, so every product loses up to one LSB, always downward (section 1).
held-out set
The four problems that decide which tableau a cell keeps: pendulum, dc_motor, rc_thermal and quaternion. The inner optimizer scores on the search set alone, but elites are selected on held-out error and the outer-loop model is shown one held-out figure per occupied cell, so a champion's held-out error carries selection bias and is not a clean test-set result.
hypothesis ledger
An append-only file of falsifiable statements about the archive, each carrying a machine-checkable predicate over cell statistics. Code writes the verdict once the named cells have enough records.
L-stability
A-stability plus a stability function that tends to zero as stiffness grows, so a very fast mode is damped out in one step. For the two-stage SDIRK it holds only for particular values of the diagonal gamma.
LSB
Least significant bit: the smallest step Q15 can represent, 2^-15, about 3.05e-5.
MAP-Elites archive
One grid per algebraic order 1 to 4, with cells keyed by stage count 2 to 6 and cost bucket 0 to 7, each holding only its elite. The filled grid, a map of the accuracy each shape and cost can reach, is the project's output.
off-archive
Measured outside the scored path: no verifier run, no archive record, and no part of any archive statistic or hypothesis verdict. Implicit and adaptive numbers are off-archive because the record schema cannot hold them; they are not order-verified and not scored, and they do not compare with an archive score.
order (measured vs algebraic)
Algebraic order is the largest p whose order conditions the exact coefficients satisfy, and it keys the grids. Measured order is the fitted slope of section 2, and the two can differ.
Q15
Signed 16-bit fixed point: an integer q in [-32768, 32767] standing for q / 32768, in which every problem state is stored. Overflow is never saturated; the verifier turns it into a rejection.
SDIRK
Singly diagonally implicit Runge-Kutta: A is lower triangular with one repeated value gamma on the diagonal, so each stage is solved for, here with a fixed number of Newton iterations. The repeated diagonal lets one Jacobian factorization serve every stage.
stage
One derivative evaluation inside a step: an s-stage method calls the right-hand side s times per step. Stage count is one axis of the archive grids.
step-size controller
The rule that turns an error estimate into the next step size, here proportional-integral on the ratio of tolerance to estimated error. A step whose estimate exceeds the tolerance is rejected and retried smaller, so rejected attempts cost work too.
stiffness ratio
The fastest decay rate in a problem over the slowest, from its linearization. A large ratio caps an explicit method's step size at a mode that has already died away, and the validation suite labels problems stiff on this basis.
tableau
The Butcher tableau (A, b, c) that defines a Runge-Kutta method: stage weights, final weights and stage times, with A strictly lower triangular for an explicit method. On this site every tableau is exact fractions, hashed by content.
tier names
The evidence tier a record gets on entering its cell by the rules in section 3: heldout_verified, search_only, no_incumbent for an empty cell, no_improvement when an incumbent was there and neither applied, and unreplicated, the one word those last two replaced, which no record carries now. The grid ranks on held-out error alone, so the heldout_verified count can fall without anything going wrong.
verifier hash
A sha256 over the ten pinned scoring files, checked at container start and stored on every record. Every score therefore carries the version of the code that produced it, and a table holding more than one hash says so beneath it.
work-precision
A chart of achieved error against work spent, one line per method, for comparing methods that share no control parameter. Down and to the left is better.

References

  1. rk_harness/fixedpoint.py; HANDOFF 4.2 (Q15 semantics) and 10 (divergence vectors).
  2. rk_harness/coeffrep.py; HANDOFF 4.2b (coefficient representation, CSD weight).
  3. rk_harness/problems.py and fixtures/problems.json; HANDOFF 11 (problem fixture).
  4. rk_harness/simulate.py (Q15 and float64 integrators, derivative scaling, budget-to-steps).
  5. rk_harness/evaluator.py; HANDOFF 4.7 (equal-budget rule, measured order, stability extents).
  6. rk_harness/verifier.py; HANDOFF 4.4 (nine ordered checks).
  7. rk_harness/costmodel.py, fixtures/known_sequence.s; HANDOFF 4.5 (counting rules), 9.5 (anchor comparison) and 12 (assembly fixture).
  8. rk_harness/orderconditions.py; HANDOFF 4.3 (rooted trees, dyadic impossibility).
  9. rk_harness/search.py; HANDOFF 4.9 (CMA-ES, projection, islands).
  10. rk_harness/enumeration.py; HANDOFF 8 (phase table) and 9.6 (phase 0 count).
  11. rk_harness/archive.py; HANDOFF 4.8 (grids, buckets, tiers, replay).
  12. rk_harness/ledger.py; HANDOFF 6 (predicate grammar, verdicts, effect size).
  13. rk_harness/quarantine.py; HANDOFF 7 (admission checks, shadow mode, proposal gate).
  14. rk_harness/verifier_hash.py, VERIFIER_HASH, rk_harness/credentials.py; HANDOFF 4.11 and 2.2.
  15. rk_harness/directive.py, rk_harness/runner.py; HANDOFF 5 (directive schema and validation).
  16. entrypoint.sh; HANDOFF 13.1 (start-up order: probe, hash, golden gate).
  17. tests/test_t1_fixedpoint_coeff_cost.py through tests/test_t7_methodology.py; HANDOFF 14 (acceptance criteria G, F, V, C, K, R, E).
  18. scripts/preflight.py and docs/REVIEW-REPORT.md (executed review checklist).
  19. rk_harness/falsification.py; HANDOFF 15 (criteria and sweep).
  20. rk_harness/sitegen.py; HANDOFF 17 (auto-publish rules, determinism, labels).
  21. docs/HANDOFF.md (frozen specification; section numbers as cited above).
  22. M. Croci and M. B. Giles, "Effects of round-to-nearest and stochastic rounding in the numerical solution of the heat equation in low precision", IMA Journal of Numerical Analysis 43(3), 2023, pages 1358-1390.
  23. M. Hopkins, M. Mikaitis, D. R. Lester and S. Furber, "Stochastic rounding and reduced-precision fixed-point arithmetic for solving neural ordinary differential equations", Philosophical Transactions of the Royal Society A 378:20190052, 2020.
  24. M. Croci and G. Rosilho de Souza, "Mixed-precision explicit stabilized Runge-Kutta methods for single- and multi-scale differential equations", Journal of Computational Physics 464, 2022, 111349.