Annex: the table the gift sits on
Everything the survey rests on, served as files rather than described: the preregistration, both implementations, the full results transcript, the integrity records, the calibration materials, the ledger, and the corrections we made along the way. Nothing is held back.
Each file below is byte-identical to what was produced during the run. The arm deliverables carry the same sha256 they were hash-stamped with at lock, so you can confirm from the ledger that what you are reading is what was attested — not a later edit.
The method, as declared
| file | what it is | bytes |
|---|---|---|
| PREREG | the method, written before any comparison, with its additions log | 6,813 |
| PREDRAFT | both directions pre-drafted before the run — what would be reported either way | 2,770 |
| RECON | what was looked for in the publisher’s own surfaces, and what could not be found | 6,937 |
The instruments
| file | what it is | bytes |
|---|---|---|
| impl_A.py | first implementation — pandas + numpy | 20,156 |
| impl_B.py | second implementation — pyarrow read, pure-Python aggregation, no shared code | 18,417 |
| compare_impls.py | the comparator that checks the two against each other | 1,839 |
| source_hash.txt | the shipped source hash the implementations read | 64 |
What they produced
| file | what it is | bytes |
|---|---|---|
| RESULTS | the full transcript: all 1,292 cell classifications, every declared policy including those that did not fit, per-column permutation nulls | 65,951 |
| REPORT | the report, with the preregistration reproduced verbatim inside it | 36,288 |
Checks run by contexts that did not do the work
| file | what it is | bytes |
|---|---|---|
| GATE (mechanical) | script-checked items — deliverables, ordering, both implementations, process determinism, nulls, input hashes, skeleton, isolation. Content-free by construction, so it can be run without seeing any result | 2,864 |
| INTEGRITY (reading) | the items a script cannot judge: cell-census completeness, whether the two implementations are genuinely independent, null coverage, phrasing rules — published with its qualifications intact | 8,415 |
| PRIMING | earlier iterations of this study were sealed away so this run could not see them; this is the check, by a separate context, of whether the independent result was shaped by prior knowledge | 15,269 |
Calibration
Two positive controls, each predicted and hash-stamped before the perturbed corpus existed. A permutation null shows a measured deviation is not chance; it cannot show the instrument would have seen a deviation had one been there. These do.
| file | what it is | bytes |
|---|---|---|
| score control (prereg) | three resolved bits flipped; predicted -0.600000 | 277 |
| cost control (prereg) | one instance cost raised by 10.00; predicted +0.15625 and +0.03125 | 428 |
| results | predicted against measured, both implementations, both paths | 2,125 |
| baseline A | unperturbed run, impl_A — byte-identical across both controls, run days apart in separate directories | 314,675 |
| baseline B | unperturbed run, impl_B | 314,542 |
| score pert A | after three resolved bits flipped, impl_A | 314,676 |
| score pert B | the same perturbation through impl_B | 314,542 |
| cost pert A | after one instance cost raised by 10.00, impl_A | 314,659 |
| cost pert B | the same perturbation through impl_B | 314,525 |
| retest_cost_denominator.py | reproduces the addendum's denominator comparison from the two parquets | 4,192 |
Provenance and terms
| file | what it is | bytes |
|---|---|---|
| ADDENDUM | two surfaces examined after publication — the Space the recon missed, and the extractor run late; includes the finding that per-instance costs are normalised toward the published figure | 7,016 |
| extract_instance_results.py | the publisher’s extractor — cited, not copied: read it at
OpenHands/openhands-index-results commit 3015ac6 and check sha256
4779a4c7f0c4063655c37fdd65b1f15b8e8f20f544a64bdfb98753c1da09821e
against what you receive. Their code, their licence to state | — |
| LEDGER | the append-only record of what was hashed and when, including the notes recording our own process failures. Paths reduced to basenames; hashes untouched | 6,813 |
| MANIFEST | sha256 and size of every file in this directory | 3,179 |
| REPRODUCE | the no-network path, start to finish — run in a clean directory before it was written | 2,709 |
| LICENSE | MIT for the implementations, CC BY 4.0 for the documents. The method is not held as proprietary; anyone may run it, adapt it, or turn it against us | 2,200 |
Corrections
Recorded because a method is worth what its errors are worth.
A sweep that mis-stated the reconciliation rate. An early custodian pass reported that 169 of 170 score cells reconciled — the 170 being the five per-category score columns across 34 rows, not the 204-cell score family, which also includes the average. It had conflated stable under both missing-value policies with reconciled, so cells failing under both were counted as agreeing. The locked census — 340 reconciled, 102 divergent — is the authority, and the sweep’s figure is withdrawn.
A flagged cell that was not a finding. The same sweep flagged
issue_resolution_score for one model as divergent. Both implementations classify
it as reconciled within the declared 0.5 tolerance — one of 14 score cells reconciled at that
tier, not an outlier. Checked, dismissed, and the dismissal is in the ledger.
A preregistration hash taken late. The intended mid-run attestation did not fire; the custodian was watching for a file-write prompt the run never surfaced. The preregistration’s content is fixed from lock forward, but not from declaration forward. For this arm the ordering evidence is filesystem timestamps — preregistration 18:49:33, first computation output 18:54 — rather than a declaration-time hash. Stated because it is weaker than intended.
A checker that reported three false failures. The mechanical script assumed
a flat workspace and a data/ directory; this arm used a work/ layout,
so it mis-read three items. Diagnosed as script artifacts and confirmed by the reading check,
not silently suppressed. The gate output above is published as it ran, false failures included.
What was designed and not run
A second executor on separate hardware, and a blinded pair of arms, were specified and not executed. The study closes at one executor plus calibration. The reproduction path is the open second seat, and the most useful replication would be the publisher’s own.