# ADDENDUM — two surfaces examined after publication **2026-07-26, custodian-side. Post-lock: nothing here alters the arm's deliverables, its census, or its hashes. It records surfaces the recon did not examine, what they showed, and one finding that weakens a result we published earlier the same day.** Prompted by a reply from a maintainer, who pointed out that the project's code is open and asked, reasonably, why the values could not simply be recomputed. --- ## 1. The Space — a surface R3 missed `huggingface.co/spaces/OpenHands/openhands-index` was **not examined** by the arm's recon. R3 read the dataset card, `publish_hf_dataset.py`, and the results-repo README, and recorded what it looked for and could not find. It did not look here. That is a gap in the recon, and it is ours. What the Space shows, read 2026-07-26: - `simple_data_loader.py` builds each category score as the arithmetic mean of the per-benchmark `score` values read from `results//scores.json`, and each category cost as the mean of `cost_per_instance` from the same file. - Neither the loader, `leaderboard_transformer.py`, nor `setup_data.py` reads the instance table at any point. The only occurrence of the string "instances" in the loader is a docstring describing class instances. - `DATA_STRUCTURE.md` documents the *shape* of `scores.json` — `score`, `metric`, `cost_per_instance`, `average_runtime`, `full_archive` — and does not state how `score` is produced from an archive. **Effect on the published result: none, and it corroborates.** The survey's account — that the leaderboard and the instance table are siblings descending from the eval archives, with neither computed from the other — is what the Space's own code does. The published `score` arrives already computed, carrying a metric label. Open source code and independently recomputable published values are different properties; the project has the first, and for two of its categories it also has the second. ## 2. `extract_instance_results.py` — an unexecuted leg, now run Recorded in RECON as present-but-not-fetched. Fetched 2026-07-26 at the same source commit `3015ac612e7196f428e6e8a3948965d32d9a3331`, 23,911 bytes, sha256 `4779a4c7f0c4063655c37fdd65b1f15b8e8f20f544a64bdfb98753c1da09821e`. It is the publisher's file, not ours, and it is **cited rather than copied**. The repository declared no licence in its API metadata at the time of reading while its README named MIT — reported as-is in RECON — so redistributing it here would assert a permission we cannot point to. Read it at the source and check the hash above against what you receive: https://raw.githubusercontent.com/OpenHands/openhands-index-results/3015ac612e7196f428e6e8a3948965d32d9a3331/scripts/extract_instance_results.py That is stronger than a mirror in any case: it asks you to trust the publisher's own bytes rather than our copy of them. It is an extractor. It computes no score, rate, or denominator. Two things in it bear on the survey: **(a) Where the nulls come from.** `TERMINAL_STATUSES` maps `resolved` → true and `unresolved`, `invalidated`, `error`, `incomplete`, `empty_patch` → false. A status outside that map yields `resolved: null`. This is consistent with the dataset card and with the convention the survey derived; it is now sourced rather than inferred. **(b) Per-instance costs are normalised toward the published figure.** `_normalize_costs_to_target_mean(records, ref.cost_per_instance)` computes the mean cost over *graded* records — those with both `resolved` and `cost` non-null — and scales **every** costed record by `published_cost_per_instance / current_mean`. Where no graded record carries a cost, the published figure is assigned directly. ## 3. What (b) costs us **The cost columns cannot serve as an independent check, in either direction.** The per-instance costs shipped in `instances.parquet` have been scaled toward the very figure the survey compared them against. Where a cost column reconciled, that agreement is partly constructed rather than independently derived — a weaker result than the survey implied by treating score and cost columns alike. This was tested, not assumed. Recomputing every per-category cost cell under the script's own population — graded records only — and classifying against the arm's declared tolerance tiers: | population, **under the primary cost policy** | reconciled / 170 | T0 | T1 | T2 | divergent | |---|---|---|---|---|---| | the survey's (cost non-null) | 115 | 29 | 37 | 49 | 55 | | the script's (graded records) | 114 | 33 | 33 | 48 | 56 | **Both rows hold the policy fixed**, in order to isolate the denominator. Neither is the census figure. The census rule counts a cell reconciled if *any* declared policy reproduces it, which for these same 170 cells gives **142** — the number the per-column table on the survey page implies. The retest asks a narrower question than the census does, and does not re-run it. The script that produces this table is published as `retest_cost_denominator.py`; it needs the two parquet files and nothing else. **Zero cells move from divergent to reconciled.** The census published in the survey stands unchanged under its own rule; the divergences are not an artifact of the denominator we chose. Neither population recovers the published figure exactly, which is consistent with the shipped sidecars being a partial rendering of the archive population the published cost was computed over. The score columns are unaffected: nothing in the extractor scales or targets them, and the two categories certified as exact remain independently derived. ## 3a. Which rows the controls perturbed Both positive controls were run on Claude rows, and the score control corrupted this survey's own executor: `OpenHands/claude-fable-5` on `swe-bench` for the score path, and `OpenHands/claude-opus-4-7` on `swe-bench-multimodal` for the cost path. Neither was chosen — both fell out of a declared seed, recorded in the preregistration files before either run. A control demonstrates that the arithmetic responds at a predicted magnitude; it does not depend on which row is perturbed. Stated because the coincidence is visible in the published preregistrations and is better named than found. ## 4. What still is not established The definition of `score` inside `scores.json` — the denominator used, and the treatment of instances with no terminal status — is upstream of every public surface examined here. It is what would let a third party verify the remaining categories, and it is not in the dataset bundle, the publishing script, the results-repo README, or the Space. ## 5. Status The arm's preregistration, results, report and census are unchanged and their hashes still verify. This addendum is custodian-side work performed after the arm locked, and is marked as such. The certificate is unchanged: it certifies two score columns and explicitly certifies nothing about cost.