Reproducibility is a build system problem

We have a rule in the lab: a result that cannot be reproduced from a clean checkout is not a result. Every track has to satisfy it before anything gets written up.

Applying that rule taught us something we did not expect. When a rebuild fails, the problem is almost never the method. It is the environment.

The actual causes, ranked

From our own failed rebuilds over the past year:

1. Unpinned transitive dependency. The direct dependencies were pinned. Something three levels down was not, it released, and behaviour changed.
2. Unseeded randomness. Not in the obvious place. In a data loader's shuffle, or a library's internal tie-breaking.
3. Hardware difference. Different GPU, different kernel selection, different floating-point accumulation order, different answer in the last digits — which matters if anything downstream thresholds on it.
4. Uncommitted local state. A config file that lived on one laptop. The classic.
5. Actual methodological error. Real, and rare, and vastly outnumbered by the four above.

Four out of five failed rebuilds were infrastructure. If you want reproducible research, most of the work is engineering, and it is unglamorous.

What we require now

A lockfile, and a build that fails without it. Not a requirements list — a full transitive lock with hashes. If the lockfile is missing, the build refuses rather than resolving fresh.

One seed, threaded everywhere. A single seed at the top that reaches every source of randomness, including the ones in libraries you did not write. Finding them all is tedious. Do it once.

Recorded hardware, and tolerances that acknowledge it. We log the device a result came from and state numerical tolerances explicitly. A result that only holds to the last bit on one specific GPU should say so.

Rebuild on a schedule, not on demand. This is the one that actually works. Every published result rebuilds weekly in CI from a clean checkout. When it breaks we find out within a week, while somebody still remembers the context.

# .github/workflows/rebuild.yml
on:
  schedule: [{ cron: "0 6 * * 1" }]
jobs:
  rebuild:
    strategy:
      matrix: { result: [swarm-orchestrator, cortex-bridge, twin-bench] }
    steps:
      - uses: actions/checkout@v4
      - run: make verify RESULT=${{ matrix.result }}

The part that is genuinely hard

Some of our results depend on physical hardware — a specific EEG headset, a specific board, a specific room. No lockfile captures that.

The best we have managed is to record the calibration state alongside the result and to state plainly which numbers are hardware-conditional. It is not reproducibility. It is an honest description of what would be required, which is the next best thing and is more than most write-ups offer.

Why we bother

Partly integrity. Mostly self-interest. A result you can rebuild is a result you can extend, hand to a new student, or use as a baseline nine months later. Every hour spent on the harness has come back several times over.