Certificate of Independent Recomputation

Two columns of the openhands-index leaderboard were recomputed from the shipped instance table alone, under a method preregistered before any comparison was run.

What is certified

Aggregation conventions used, derived from the data — the publisher documents what the nulls mean, the aggregation of them is this survey's derivation: instances with no recorded outcome are counted in the denominator and not as resolved; instances with no recorded cost are excluded from the cost mean.

How it was verified

Method preregistered to disk before computation and reproduced verbatim in the report. The hash-attestation in the append-only ledger was taken at lock rather than at declaration — later than intended; the ordering evidence is filesystem timestamps, set out in the annex corrections. Two implementations sharing no code — pandas/numpy and pyarrow/pure-Python — agreeing cell for cell across all 1,292 cells. Each implementation run as two separate OS processes, outputs byte-identical. A permutation null for every measured column, seed 20260725, 1,000 draws, present in the results transcript beside each measurement. Complete cell accounting: every one of 34 × 38 = 1,292 cells classified, with both implementations producing identical classifications.

Instrument sensitivity demonstrated by declared positive control: three instance outcomes flipped on a copy, predicted deviation -0.600000 stamped to the ledger before the perturbed run existed, measured -0.600000 in both implementations. A second control on the cost path — one instance cost raised by 10.00, predicted +0.15625 — returned that figure exactly in both implementations. Both aggregation paths are calibrated.

Measured blind, from the shipped bundle alone. The publisher's own source — the Space that renders the leaderboard, and the script that extracts the instance sidecars — was read afterwards at the pinned source commit, and contradicts nothing certified above.

What this certificate does not certify

Any column not named above. Frontend, testing and the average columns did not reproduce for every row under any preregistered policy — 93 of the 102 divergent cells fall in those three families, and 9 fall elsewhere (information_gathering_score 2, information_gathering_cost 5, issue_resolution_cost 2). The publisher's pipeline builds the leaderboard from per-model results/<model>/scores.json and the instance table from results/<model>/instance_results/<benchmark>.json sidecars, published in lockstep from one commit, with neither computed from the other — verified in publish_hf_dataset.py from openhands-index-results at commit 3015ac612e7196f428e6e8a3948965d32d9a3331, sha256 8f125116bddca13f5d767bbec691811057e2fe9a168de942e9dc64cb41839d39, where the two builders' call graphs are disjoint. Recomputability from the instance table is therefore not a property either table was built to have. 21 of 38 columns have no antecedent in the instance table and were never candidates.

It does not certify the correctness, quality or fitness of any published value. This is a statement about recomputability from shipped bytes, and nothing else. It says nothing about data the publisher holds and did not ship.

Scope of execution

One executor — a single Claude session, model claude-fable-5 — on one machine, an Apple M4 running macOS 26.2. A no-network reproduction path is published in the annex; any reader running it constitutes the independent second execution this certificate does not itself claim.

Conflict disclosure

The executor is itself a row of this corpus — OpenHands/claude-fable-5 — as are six other models from the same vendor. The method was preregistered before any value was computed; every row received identical mechanical treatment; publication of the outcome was committed in writing before execution, whatever the outcome.

← Survey · Annex