# REPORT — Arm A2-id — 2026-07-25
**Study:** to what extent can the values published in one artifact of
`OpenHands/openhands-index` be recomputed from another artifact of the same dataset
(HANDOFF v3, Envelope 1). **Arm:** A2-id (identified corpus, Executor A2 = Claude,
fresh session, model id `claude-fable-5`). **Register:** S3 COMMUNITY. No sends, no
commits, no pushes, no contact (L8).

**Failure first (L6):** the declared method does not fit everywhere it was pointed.
No single denominator/missing-value policy fits every addressed column (anticipated
and permitted under P5, but real): the micro-average policies fail for every row of
both `average_score` and `average_cost`; the median and sum-over-resolved cost
policies fail almost everywhere; the fraction-scale score policies fail everywhere
except score 0.0/one boundary row; and for `average_score` the closest declared
policy reproduces the published value at published precision for only 6 of 34 rows.
21 of the target's 38 columns have no antecedent in the source artifact at all, so
for the majority of the target's surface the study question is answerable only as
"never a candidate." Everything below is bounded by that.

---

## 1. Environment

Shared MacBook Air (`Mac16,13`, Apple M4), macOS 26.2 (build 25C56), arm64, host
`MacBook-Air.local`. **Shared machine; this arm has no machine independence (§3.6)
— stated, not claimed otherwise.** Conda env `habitat` at `/Applications/anaconda3`
(P1): Python 3.11.13, numpy 1.26.4, pandas 2.3.2, pyarrow 14.0.2.
*verified-against-bytes* (session command output).

Workspace disclosure: the arm directory arrived pre-staged by the custodian with raw
public fetches (corpus bytes, HTTP header captures, HF/GitHub API captures, Space
code). I verified the corpus bytes by independent re-fetch (byte-identical, same
sha256) and encountered no prior-analysis material in the staging.
*verified-against-bytes*

## 2. Corpus

`OpenHands/openhands-index` (Hugging Face, Apache-2.0, ungated), main =
`db6504dfbb2ac2c905c71f73f430bf9cfdc55018` (2026-06-30), built per its card from
`openhands-index-results` commit `3015ac612e7196f428e6e8a3948965d32d9a3331`.
Files consumed (P2; each matched Appendix A where Appendix A lists it, and matched
between custodian staging and my fresh fetch):

- `test.parquet` — 44,323 bytes, sha256 `966544ad…d827a`, 34 rows × 38 columns
  (config `default`, leaderboard; one row per model). *verified-against-bytes*
- `instances.parquet` — 435,757 bytes, sha256 `f456e937…968a5`, 40,643 rows × 9
  columns (config `instances`; one row per model × benchmark × instance;
  `resolved` bool with 475 nulls, `cost` double with 436 nulls).
  *verified-against-bytes*
- `README.md` — 5,226 bytes, sha256 `4e134a2f…b60fb`. *verified-against-bytes*
- shipped `.source_hash` = `886a3ebabd70879b712deb36fcfc190199e9c91a0ea538a89b156a20663fda29`.
  *verified-against-bytes*

Encoding artifacts, named before any magnitude that touches them (L5/D3):
- `resolved` nulls and `cost` nulls are properties of the sidecar encoding — the
  publisher's own card says `resolved` is "null when the archive didn't record an
  outcome" and `cost` "null when unavailable." Any denominator difference downstream
  of those nulls is a property of the encoding choice, not a discovery.
- `categories_completed` is 5 for all 34 rows (zero variance): its uniform match is
  schema-population, carries no discriminating information, and is classified
  structural, not reconciled.
- `agent_name` and `agent_type` are single-valued across both artifacts (the card
  itself notes `agent_type` is "currently always OpenHands"): structural.
- `id` is the join key itself: its self-match is schema-forced, structural.

The full R-series recon (org enumeration, revision history, publisher statements,
license, Appendix A re-verification) is filed as `RECON_A2-id_2026-07-25.md`.
The publisher's pipeline (`publish_hf_dataset.py` at the pinned source commit)
states the leaderboard config is built from `results/<model>/scores.json` and the
instances config separately from `instance_results/*.json` sidecars; **the
publisher nowhere states that the leaderboard values are computed from the instance
table.** Correspondence between the two artifacts is therefore tested below, not
presupposed (O6). *verified-against-bytes* (script at pinned commit)

## 3. Declared method (preregistration, reproduced verbatim per L1)

The complete preregistration file follows, unmodified, including its additions log.

---
# PREREGISTRATION — Arm A2-id
**Study:** openhands-index recon/discovery/lawful run (HANDOFF v3, 2026-07-24)
**Arm:** A2-id — identified corpus, Executor A2 (Claude, fresh session; model id `claude-fable-5`)
**Date written:** 2026-07-25 (US/Eastern), before any reconciliation computation.
**Status at time of writing:** Recon R1–R5 executed; schema-only inspection of both
parquets executed (column names, dtypes, row counts, null counts, decimal-precision
census). **No cross-artifact join, aggregation, comparison, or reconciliation has been
computed at the time this file is written.**

---

## 0. What was inspected before this preregistration (disclosed per L1)

- Bytes fetched and pinned (sha256, sizes, HTTP status, `x-repo-commit`) for
  `test.parquet`, `instances.parquet`, `README.md` of `OpenHands/openhands-index`
  at main = `db6504dfbb2ac2c905c71f73f430bf9cfdc55018`.
- The dataset README (a publisher claim, to be tested, not assumed).
- Publisher pipeline code `scripts/publish_hf_dataset.py` at source commit
  `3015ac612e7196f428e6e8a3948965d32d9a3331` of
  `github.com/OpenHands/openhands-index-results` (R3 surface; its statements are
  claims entering the policy enumeration below, not privileged truths).
- Schemas: `test.parquet` = 34 rows × 38 columns; `instances.parquet` = 40,643 rows ×
  9 columns (`id, agent_name, agent_type, language_model, benchmark, category,
  instance_id, resolved (bool, 475 nulls), cost (double, 436 nulls)`).
- Decimal-precision census of `test.parquet` float columns (formatting property):
  per-category `*_score` max 1 decimal; `average_score` max 2; all `*_cost` max 4;
  `average_runtime` max 2; per-category `*_runtime` max 0–1.
- The workspace arrived pre-staged by the custodian with raw public fetches (parquets,
  HTTP headers, HF/GitHub API captures, Space code). I verified the parquet bytes by
  independent re-fetch (hashes identical). No prior-analysis material was encountered.

No value of any `test.parquet` cell has been compared to any recomputation at this
point. The precision census and null counts above are the only value-derived facts
consumed pre-registration, and they are used solely to set tolerance units (P6).

## 1. Designated recompute target (P4/P12)

**Target artifact: `test.parquet` (config `default`), all 34 rows × 38 columns =
1,292 cells.** Source artifact for recomputation: `instances.parquet`. Direction:
recompute `test.parquet` values from `instances.parquet` alone (no network, no other
inputs in the recompute path — O9).

## 2. Correspondence derivation (P4)

### 2.1 Row mapping (join key) — derived by search, not assumed
Candidate join keys, all to be evaluated; winner = highest (coverage × uniqueness);
runner-up reported:
- K1: `test.id` = `instances.id`
- K2: `test.language_model` = `instances.language_model`
- K3: (`test.agent_name`, `test.language_model`) = same pair in instances
- K4: case-insensitive variant of K1

Coverage = fraction of test rows with ≥1 instance row under the key; uniqueness =
no test row maps to instance rows that also map to another test row. Both directions
reported (test rows without instance groups; instance groups without test rows).

### 2.2 Column-family mapping (benchmark → per-category columns)
Candidate mappings from `instances.benchmark` to test column prefixes:
- C1: the (benchmark, category) pairing observed inside `instances.parquet` itself,
  slugged `category.lower().replace(' ', '_')`.
- C2: the README table pairing (claim).
- C3: the publisher-code constant `BENCHMARK_TO_CATEGORY` (claim).
All three evaluated for consistency; disagreements reported. If C1 is internally
inconsistent (one benchmark ↔ multiple categories), that is reported and C1 decides
per majority with the tie reported.

### 2.3 Column correspondences and no-antecedent declarations
Addressed columns (candidate antecedent exists in `instances.parquet`):
- Identity/categorical: `id` (join key itself — classified structural if used as key),
  `agent_name`, `agent_type`, `language_model` — antecedent = the set of values in the
  model's instance rows.
- `<cat>_score` ×5 — antecedent = `resolved` over the model×benchmark rows.
- `<cat>_cost` ×5 — antecedent = `cost` over the model×benchmark rows.
- `average_score`, `average_cost` — antecedent = compositions of the above.
- `categories_completed` — antecedent = benchmark presence per model.

No-antecedent declarations (no column or combination in `instances.parquet` carries
the quantity; reason in parentheses):
- `sdk_version`, `openness`, `country`, `supports_vision`, `release_date`
  (metadata.json provenance per publisher code; nothing in instances schema).
- `average_runtime`, `<cat>_runtime` ×5 (no runtime column in instances).
- `<cat>_logs_url` ×5, `<cat>_visualization_url` ×5 (no URL columns in instances).
Total no-antecedent columns: 21; addressed columns: 17.

## 3. Policy enumeration (P5) — declared exhaustively, all reported

Notation: for a model m and benchmark b, G(m,b) = instance rows joined under the
winning key and column mapping. R⁺ = count(resolved == True), N_all = |G|,
N_obs = count(resolved non-null), N_cost = count(cost non-null).

### 3.1 Per-category `<cat>_score` (5 columns)
Aggregation family: proportion of resolved, scaled.
- Denominator policies: **D_all** = N_all; **D_obs** = N_obs.
- Missing-value policies (on `resolved`): **M_false** (null → not resolved; pairs with
  D_all) and **M_excl** (null excluded; pairs with D_obs). The pairing is forced by
  coherence; both pairs are computed: S1 = 100·R⁺/N_all, S2 = 100·R⁺/N_obs.
- Scale candidates: percent (×100) and fraction (×1) — both computed.
- Rounding candidates: none; round-half-even to 1 dp (published precision);
  round-half-even to 4 dp (publisher-code constant).
- Conditioned candidates (P5 conditional clause): none beyond the b-conditioning that
  defines the column family; no further structure derived under P4 admits more.
**Primary policy (for P9 nulls and narrative ordering only; all policies reported):
S1_pct** = 100·R⁺/N_all, unrounded comparison plus round-to-1dp comparison.

### 3.2 Per-category `<cat>_cost` (5 columns)
- Aggregation candidates: **mean**, **median**, **sum/N_all**, **sum/N_resolved⁺**
  (sum of non-null costs over count of resolved==True).
- Missing-value policies (on `cost`): **M_excl** (non-null only), **M_zero**
  (null → 0.0, denominator N_all).
- Denominator policies: N_cost vs N_all (these instantiate the two missing policies
  for the mean; for median, M_excl only — a null-as-zero median is also computed for
  completeness).
- Rounding: none; round-half-even to 4 dp.
**Primary: mean over non-null costs (M_excl), unrounded + 4dp.**

### 3.3 `average_score`
- **AS-macro-raw**: mean of the 5 per-benchmark recomputed scores (primary per-cat
  policy), then round-half-even to 4 dp (publisher-code form) and to 2 dp.
- **AS-macro-pub**: mean of the 5 per-benchmark scores first rounded to 1 dp.
- **AS-micro**: 100·ΣR⁺/ΣN over all the model's rows (both D_all and D_obs).
- Denominator variants: over benchmarks present (categories completed) vs over all 5
  (missing benchmark → excluded vs → 0); both computed.
**Primary: AS-macro-raw over benchmarks present, D_all.**

### 3.4 `average_cost`
- **AC-macro-raw**: mean of the 5 recomputed per-benchmark mean costs (M_excl).
- **AC-macro-pub**: same with per-benchmark values rounded to 4 dp first.
- **AC-micro**: mean over all the model's non-null costs; plus M_zero variant.
- Denominator variants: benchmarks present vs all 5; both computed.
**Primary: AC-macro-raw over benchmarks present.**

### 3.5 `categories_completed`
- **CC1** = count of distinct benchmarks with ≥1 instance row for the model.
- **CC2** = count of distinct benchmarks with ≥1 non-null `resolved` row.
Integer comparison, exact only.
**Primary: CC1.**

### 3.6 Categorical columns (`agent_name`, `agent_type`, `language_model`)
- **U-unique**: reconciles iff the value set over the model's instance rows is a
  singleton equal to the test value.
- **U-major**: reconciles iff the modal value equals the test value; multiplicity of
  the value set reported.
**Primary: U-unique.**

No single policy fits every column (permitted statement under P5): counts and
categoricals admit no denominator/missing grid; this is stated rather than forced.

## 4. Tolerance tiers (P6) — declared now, in published units, not adjusted later

Score columns (percent points): **T0** = equality after rounding recomputed value
half-even to the column's published precision (1 dp per-category, 2 dp average);
**T1** = |Δ| ≤ 0.05; **T2** = |Δ| ≤ 0.5; else **divergent** (under every declared
policy → cell class "divergent"). Δ measured on unrounded recompute vs published.
Cost columns (USD): **T0** = equality at 4 dp; **T1** = |Δ| ≤ 0.005; **T2** =
|Δ| ≤ 0.05; else divergent.
`categories_completed`: exact integer equality only (T0 or divergent).
Categorical: equality or divergent.
Per-column and per-row deviations reported, plus count of rows matching at each
column's own published precision (P6), for every declared policy, not only primary.

## 5. Independent implementations (P7) and determinism (P8)

- **Impl A**: Python + pandas (+pyarrow parquet reader via pandas).
- **Impl B**: Python + pyarrow only for byte reading, all aggregation in pure-Python
  dict/loop code; no pandas import; written after A without referring to A's code
  beyond this preregistration. Shared inputs: the two parquets and this file only.
- **Agreement threshold (declared):** for every (row, column, policy) recomputed
  value: both null, or |a−b| ≤ 1e-9 absolute; every cell classification identical;
  every match count identical. Anything less fails the P7 gate and is reported.
- **P8:** each implementation executed as two separate OS process invocations;
  canonical output files (sorted keys, fixed float repr `repr(float)`, `\n` EOL)
  byte-compared with `cmp`; diff results recorded in RESULTS.

## 6. Negative control (P9) — permutation null

- Scheme: permute the assignment of the 34 join-key groups of `instances.parquet` to
  the 34 `test.parquet` rows (uniform random permutation of group labels).
- Seed: `numpy.random.default_rng(20260725)`; draws: **1000**; identical scheme in
  both implementations is NOT required (P9 lives in the RESULTS transcript from
  Impl A; Impl B reproduces the observed statistics, and the null is additionally
  re-run in Impl B with the same seed via numpy — numpy is permitted in Impl B for
  the null only, since the null is not a reconciliation of the corpus).
- Statistics per addressed column, under its primary policy, per draw:
  (a) count of rows reconciled at T0; (b) mean absolute deviation (numeric columns);
  (c) match count (categorical/integer columns).
- Reported per column: **median and 5th percentile** of the null distribution of each
  statistic (5th percentile of MAD is the null's favorable tail; for match counts the
  95th percentile is also recorded so the favorable tail is visible in both
  directions), plus the observed value alongside.
- Where a column's null distribution is degenerate (zero-variance column), that is a
  structural floor: stated as structural, ratios against it undefined, not infinite.

## 7. Spread (P10), rankings (L4), and integrity extras

- Every addressed column's own range (min, max) over the 34 rows is reported; a
  zero-variance column's match is flagged as carrying no discriminating information
  and the cells classified **structural**.
- Ranking: the census of ties in published `average_score` at its published precision
  is reported, and whether the published sort order (descending `average_score`) is
  decidable; the same census for the primary recomputed average. No statement about
  which records produced any movement (L4).
- **M-H (hash reconstruction, declared measurement):** recompute the publisher-code
  `content_hash` (sha256 over both tables serialized as column-sorted CSV,
  `float_format="%.10g"`, joined by `\n--\n`) from the shipped parquets and compare to
  the shipped `.source_hash` (`886a3ebabd70879b712deb36fcfc190199e9c91a0ea538a89b156a20663fda29`).
  Equal / not equal is reported either way; this tests serialization identity of the
  shipped bytes against the publisher's own no-op detector, nothing more.

## 8. Cell accounting (P12) — classification procedure

Each of the 1,292 cells receives exactly one class, by this precedence order,
declared now:
1. **no antecedent** — cell in a no-antecedent column (§2.3), or in an addressed
   column where the join yields zero instance rows for that model (per-category
   columns: zero rows for that model×benchmark).
2. **structural** — cell in an addressed column whose `test.parquet` values have zero
   variance across the 34 rows (match carries no discriminating information), or
   whose value is schema-forced.
3. **reconciled at tier N under policy P** — the best (lowest-N) tier achieved by any
   declared policy; the achieving policy named; all policies' tiers retained in the
   transcript.
4. **divergent under every declared policy** — otherwise.
Counts by class must sum to 1,292; the sum is printed in RESULTS.

## 9. Row-treatment rule (§3.5) — declared before computing

Rows receiving individual mention in the report are exactly:
(a) per addressed numeric column, the row(s) of maximum |Δ| under the primary policy,
    ties included, named mechanically;
(b) the executor-corresponding rows under §10 below (both executors, symmetric);
(c) rows involved in join failures (no instance group), if any.
No other row is discussed individually. No characterization of what produced a
residual (L2/L4).

## 10. Self-identification rule (§3.1) — declared before looking

The executing model of this arm is `claude-fable-5` (system-reported model id).
**Mapping rule:** a `test.parquet` row corresponds to an executor iff
`lower(language_model)` equals the executor's model id lowercased; if no exact
match, the fallback is substring match on the model family token (`fable` for A2).
Applied symmetrically to Executor B2 (Gemini CLI): B2's pinned model id is **unknown
to this arm** (recorded fact); therefore B2's candidate set = rows where
`lower(language_model)` begins with `gemini`, reported as a set with the note that
the pin is unknown, not resolved by choosing (P11).
Own-row values, rank, and the rule above are reported per §3.1. If the §9(a) rule
lands on an executor row, that is recorded as the §3.3 self-example-hazard outcome.

## 11. What locks this arm

This preregistration, then: PREDRAFT (both directions) → computation → RESULTS →
REPORT (containing this file verbatim, boundary section, "what this does not show").
Lock = those files on disk in `habitat_arms/iter2/A2-id/`. Envelope 2′ is requested
from the custodian only after lock. No sends, commits, pushes, or contact (L8).

## 12. Additions log (L1)

(Any addition is timestamped, withdraws nothing, and precedes the measurement it
governs.)

- **A1, 2026-07-25, before any computation ran.** For tie-set determination in §9(a)
  and for cross-implementation comparison of max-|Δ| row sets, |Δ| values are rounded
  half-even to 9 decimal places before argmax/tie grouping. Reason: makes tie sets
  well-defined across two implementations whose floating-point summation order may
  differ within the declared 1e-9 agreement threshold.
- **A2, 2026-07-25, before any computation ran.** The M-H hash reconstruction (§7)
  executes in Implementation A only. Reason: the object under test is the publisher's
  pandas CSV serialization; a pure-Python reimplementation of that serializer would
  test my CSV writer, not the publisher's. Single-implementation status is disclosed
  in RESULTS and M-H is excluded from the P7 gate.
- **A3, 2026-07-25, before any computation ran.** A cell whose antecedent group exists
  but whose every declared policy yields an undefined recomputation (no non-null
  antecedent values) is classified **no antecedent**, with the distinct reason
  "antecedent rows present, all values null" recorded in the transcript. Reason:
  "divergent under every declared policy" would otherwise mislabel cells where no
  policy produced a number at all.

---

## 4. Measurements

Everything in this section is *verified-against-bytes* (computed twice by two
independent implementations, each in two processes; RESULTS carries the transcripts).
Confidence labels are per claim; where a row cites publisher text it is
*verified-against-cited-source*.

### 4.0 Gates (P7 / P8 / P9 / P12) — all pass

- **P7:** Implementation A (pandas) and Implementation B (pure Python; pyarrow reads
  only; numpy only inside the null) agree with **0 mismatches** across every compared
  section at the declared 1e-9 threshold.
- **P8:** each implementation run twice as separate processes; outputs byte-identical
  (`cmp` clean; sha256 equal per implementation).
- **P9:** a permutation null (seed 20260725, 1000 draws, group-to-row assignment
  permuted) is present in RESULTS for every addressed non-structural column.
- **P12:** 1,292 of 1,292 cells classified; the four class counts sum exactly.

No injected control was run; the declared negative control for this arm is the
permutation null only (L3 disclosure: nothing in this study was perturbed on
purpose other than the null's label permutation, which is declared here at first
mention).

### 4.1 Correspondence derivation (P4)

All four candidate join keys attain coverage 1.0 with unique keys; the mapping
between the two artifacts is a bijection on 34 groups. Winner by declared order:
**K1 (`id` = `id`)**; runner-up K2 (`language_model`). Zero test rows without
instance groups; zero instance groups without test rows. The benchmark→category
pairing derived from `instances.parquet` itself (C1) is internally unambiguous and
agrees exactly with the README table (C2) and the publisher-code constant (C3) for
all five benchmarks.

### 4.2 Cell accounting (P12 / O10) — the shape of the answer

| class | cells | share of 1,292 |
|---|---|---|
| no antecedent (never a candidate) | 714 | 55.3% |
| structural (`id` join key; `agent_name`, `agent_type`, `categories_completed` zero-variance) | 136 | 10.5% |
| reconciled (T0 199 · T1 71 · T2 70) | 340 | 26.3% |
| divergent under every declared policy | 102 | 7.9% |

Of the 442 cells where recomputation was possible and discriminating: 76.9%
reconciled at some declared tier (45.0% at T0 = the column's own published
precision); 23.1% divergent under every declared policy. "Did not reconcile" and
"never a candidate" are kept distinct throughout (O10).

### 4.3 Per-column results (null floor first, observed second — O3)

Null = permutation null (median [5th–95th pct] of T0-match count; median MAD).
Observed = identity assignment, primary policy. "Cells" = best tier across **all**
declared policies. Ranges per P10.

| column | pub. range | null T0 med [5–95] | null MAD med | obs T0 | obs MAD | cells rec/div |
|---|---|---|---|---|---|---|
| issue_resolution_score | 34.2–95.8 | 1 [0–4] | 8.95 | 33 | 0.006 | 34/0 |
| frontend_score | 16.2–70.6 | 1 [0–3] | 13.30 | 12 | 5.76 | 12/22 |
| greenfield_score | 0.0–62.5 | 5 [2–8] | 19.13 | 34 | 0.016 | 34/0 |
| testing_score | 7.6–91.9 | 0 [0–2] | 19.24 | 15 | 1.34 | 22/12 |
| information_gathering_score | 8.5–86.1 | 1 [0–3] | 24.22 | 32 | 0.29 | 32/2 |
| average_score | 15.48–81.0 | 0 [0–0] | 14.87 | 1 | 1.23 | 13/21 |
| issue_resolution_cost | 0.0395–2.1639 | 0 [0–1] | 0.65 | 6 | 0.017 | 32/2 |
| frontend_cost | 0.19–6.92 | 0 [0–1] | 1.40 | 4 | 0.42 | 16/18 |
| greenfield_cost | 0.01–12.49 | 0 [0–1] | 3.63 | 6 | 0.45 | 34/0 |
| testing_cost | 0.0354–2.085 | 0 [0–1] | 0.56 | 6 | 0.016 | 31/3 |
| information_gathering_cost | 0.04–7.91 | 0 [0–1] | 0.88 | 7 | 0.035 | 29/5 |
| average_cost | 0.1533–5.538 | 0 [0–1] | 1.27 | 4 | 0.14 | 17/17 |
| language_model (categorical) | — | 1 [0–3] | — | 34 | — | 34/0 |

Observed T0 counts exceed the null's 95th percentile for every column above except
none — i.e., the instrument responds; the null floors are stated above alongside
every observed quantity. Zero-variance columns (`agent_name`, `agent_type`,
`categories_completed`) have degenerate nulls: structural floor, ratios undefined,
not infinite (P9).

Policies that do not fit, reported per P5/O2: fraction-scale score policies fit
nowhere (except the boundary row at 0.0); micro-averaging fits no row of
`average_score` or `average_cost`; median and sum-over-resolved cost policies fit
almost nowhere; for `average_score` the best-fitting declared policy is the mean of
1-dp-rounded per-benchmark recomputations (6 of 34 rows at T0), and the
publisher-stated form (unrounded macro-mean) fits 1 of 34 at T0. For scores, the
two denominator policies (all rows vs non-null `resolved`) produce nearly identical
fits; every score cell divergent under one is divergent under both.

### 4.4 Mechanically selected rows (§3.5 rule: max |Δ| under primary policy, ties included)

issue_resolution_score → `OpenHands/Gemini-3.1-Pro` (0.2) ·
frontend_score → `OpenHands/Gemini-3.1-Pro` (13.71) ·
greenfield_score → 11-row tie at 0.05 (exactly the 1-dp rounding boundary — an
artifact of published precision, named before its magnitude per L5) ·
testing_score → `OpenHands/GLM-5` (14.08) ·
information_gathering_score → `OpenHands/Nemotron-3-Nano` (5.5) ·
average_score → `OpenHands/Gemini-3.1-Pro` (2.75) ·
issue_resolution_cost → `OpenHands/Kimi-K2-Thinking` (0.135) ·
frontend_cost → `OpenHands/GLM-5.1` (3.31) ·
greenfield_cost → `OpenHands/claude-opus-4-5` (2.11) ·
testing_cost → `OpenHands/Kimi-K2-Thinking` (0.149) ·
information_gathering_cost → `OpenHands/Gemini-3.1-Pro` (0.281) ·
average_cost → `OpenHands/Gemini-3.5-Flash` (0.595).
No characterization of what produced any of these values (L2/L4).

### 4.5 Integrity extras

- **M-H:** the sha256 recomputed from the two shipped parquets under the publisher's
  own serialization equals the shipped `.source_hash` exactly
  (`886a3eba…3fda29`). This shows the shipped tables are the tables the publisher's
  pipeline hashed — self-consistency of the publication step; it validates nothing
  upstream of it. Single-implementation measurement (prereg addition A2).
- **Tie census (L4):** published `average_score`: 0 tied pairs among 561; 34 distinct
  values; descending sort strictly decidable; file order is that sort. Recomputed
  primary averages: also 0 tied pairs. No localization or nomination is offered
  anywhere in this report.

### 4.6 Self-identification (§3.1) — rule first, then rows

Rule (preregistered): exact match of `lower(language_model)` to the executor's model
id; fallback substring; B2 rule `startswith('gemini')` because B2's pin is unknown
to this arm (recorded, not resolved — P11).

- **Executor A2 (this session):** exact match, one row — `OpenHands/claude-fable-5`,
  file rank 1 of 34, rank by `average_score` 1 of 34. Published values of that row:
  average_score 81.0 · average_cost 5.538 · categories_completed 5 ·
  issue_resolution 95.8 · frontend 70.6 · greenfield 62.5 · testing 91.9 ·
  information_gathering 84.2 · sdk_version v1.18.1 · release_date 2026-06-09.
  Own-row cells: all five per-category scores reconciled at T0; `average_score`
  reconciled at T0 under the 1-dp-macro policy (raw macro recompute 81.0095 vs
  published 81.0); `issue_resolution_cost`, `greenfield_cost`, `testing_cost`,
  `information_gathering_cost` at T1; `frontend_cost` divergent (recomputed 6.5844
  vs published 4.39) and `average_cost` divergent (recomputed 5.9762 vs published
  5.538) under every declared policy.
- **Executor B2 (Gemini CLI, pin unknown):** four candidate rows —
  `Gemini-3.5-Flash` (rank 7), `Gemini-3.1-Pro` (rank 9), `Gemini-3-Pro` (rank 19),
  `Gemini-3-Flash` (rank 20). Their per-cell classes are tabulated symmetrically in
  RESULTS §4; each has 2–4 divergent cells among the 12 addressed numeric columns,
  as does the A2 row (2).
- **Self-example hazard (§3.3), recorded outcome:** the mechanical max-residual rule
  of §4.4 selected B2-candidate rows in 5 of 12 columns and the A2 row in 0 of 12.
  Recorded, not characterized; whether those rows are representative is itself the
  measurement.

## 5. Boundary

- **Scope of the claim surface.** 55.3% of the target's cells were never candidates
  for recomputation (no antecedent), and another 10.5% are structural. Any statement
  about "recomputability of the artifact" is a statement about the remaining 34.2%.
- **Provenance boundary.** The publisher's pipeline states the leaderboard is built
  from `scores.json` and the instance table from separately extracted sidecars. This
  study measured cross-artifact agreement; it did not and cannot measure agreement of
  either artifact with the upstream archives, which are not in the corpus.
- **Tier boundary.** T1/T2 reconciliation is proximity within declared tolerance,
  not equality. Only the 199 T0 cells match at published precision.
- **Policy boundary.** "Divergent" means divergent under the declared policy family
  (two denominators × two missing-value treatments, five cost aggregations, six
  average compositions). Policies outside that family were not run and their
  outcome is unknown (O8: left undetermined).
- **Snapshot boundary.** One revision (`db6504df…`), pinned. Nothing here speaks to
  any other revision, tag, or the current state of the leaderboard Space.
- **Design boundary (§3.6).** Same machine as any co-located arms; executor's prior
  knowledge of its own benchmark profile is not removed by anything in this arm;
  cross-arm questions (position bias, executor factor) are not computable here.

## 6. What this does not show (O11)

- It does not show that any published value is wrong, erroneous, or defective, nor
  that any process failed, nor anything about intent — cause and fault are outside
  the instrument (L2), and divergence under a declared policy family is fully
  compatible with a correct pipeline whose method lies outside that family.
- It does not show the two artifacts were intended to reconcile; the publisher's own
  description implies separate construction paths from a shared upstream.
- It does not show exact reconstruction. High T0 counts on four score columns are
  not "the artifact is recomputable," and closeness (T1/T2, low MAD, high implied
  correlation) must not be mistaken for exact reconstruction — 23.1% of evaluable
  cells reconciled under no declared policy.
- It cannot adjudicate between `scores.json`-derived values and sidecar-derived
  recomputations; the upstream archives that could are absent from the corpus.
- The `.source_hash` equality shows only that the shipped bytes are the bytes the
  publisher hashed at publication; it validates no value.
- It says nothing about position bias, blinding effects, or executor effects — those
  are §9 study-level quantities requiring all four arms.
- A reader might wrongly infer that the frontend/testing columns "contain errors":
  this report states only that under every policy in the declared family, 22 and 12
  of their 34 cells respectively did not land within ±0.5 points of the published
  values, while the permutation floor for matching is stated alongside in §4.3.

## 7. Unexecuted legs (L9)

- `scripts/extract_instance_results.py` (23,911 bytes, listed in the results repo)
  was not fetched or read; the sidecar extraction semantics were taken only from the
  publisher's READMEs.
- Per-model `scores.json`/`metadata.json` files were not fetched; recomputing the
  leaderboard from `scores.json` would answer a different question than the ask and
  was not attempted.
- No cross-revision computation (R2 was enumeration only).
- Appendix A tie censuses (A1 18/21, A3 12/21) not verifiable from bytes available
  to this arm; blinding-kit constants not verifiable without breaching workspace
  isolation.
- No injected declared control beyond the permutation null.
- M-H ran in Implementation A only (prereg addition A2).
- BLINDCHECK: not applicable (identified arm). PRIMING: deferred by design until
  every arm locks (§3.4); if this session has ended by then, the reconciliation
  context runs this arm's priming check first.
- Envelope 2′ untouched; no cross-arm tabulation; no contact with the publisher.

## 8. Conflict-of-interest disclosure (O13)

This arm was executed by a model that is a subject of the corpus it measured. The
executing model's row (`OpenHands/claude-fable-5`) is rank 1 of 34 in the published
artifact. The other executor's candidate rows (four Gemini rows; pin unknown to this
arm) rank 7, 9, 19, 20. Rows were treated symmetrically: the same policies, tiers,
nulls, and mechanical row-selection rule applied to all 34; executor rows received
individual mention only under the preregistered rules (§9(b) of the prereg).
Per §11 of the handoff, the self-finding in §4.6 (own-row values, rank, and two
own-row divergent cells) is **flagged for the custodian under the standing
publication rule of 2026-07-21 and is not discharged here** — no publication act is
taken or recommended by this arm on its own behalf. Neither audience of this report
should be able to tell which outcome this executor hoped for; both pre-drafted
directions are on disk (`PREDRAFT_A2-id_both-directions.md`), and the measured
outcome lies between them.

## 9. Suitability recommendation (O14)

**Exhibit-grade** — as a measurement of this pinned snapshot under the declared
method, with the boundary of §5 attached inseparably. Basis (mechanical): all four
mandatory gates pass (P7 zero mismatches; P8 byte-identical process repeats; P9
per-column nulls in the RESULTS transcript; P12 1,292/1,292 cells classified);
the corpus license (Apache-2.0) permits publication under L7. This recommendation
does not execute anything: study-level reconciliation (§9 of the handoff) and the
custodian's handling of the §8 flag precede any downstream use, and "not
exhibit-grade" and "undecided" were equally available had the gates or accounting
fallen short.

## 10. Open questions (phrased without presupposing any state of the data — O6)

1. What rule generates the published per-benchmark `cost` values from the upstream
   data? (Searched the publisher's README, card, and pipeline script; the
   `cost_per_instance` field is named but its construction is not stated on the
   surfaces examined.)
2. What rule generates the published `frontend_score` and `testing_score` values?
   (Both declared denominators leave 22 and 12 of 34 rows respectively outside every
   declared tier; whether some other rule fits was not tested.)
3. Do `scores.json` values and the instance sidecars for a given model descend from
   the same archive snapshot? (Not answerable from this corpus.)
4. What does `extract_instance_results.py` do at the pinned commit? (Not read.)
5. Which model id is Executor B2's pin? (Held by the custodian; needed before the
   B2 self-identification rule can resolve to fewer than four rows.)
6. Does the R2 observation that tag `v2026.06.30-3015ac6` and branch `main` target
   different commits bear on anything measured here? (Unexamined.)

---
*Deliverables of this arm:* `PREREG_A2-id_2026-07-25.md` ·
`PREDRAFT_A2-id_both-directions.md` · `RECON_A2-id_2026-07-25.md` ·
`RESULTS_A2-id_2026-07-25.txt` · this report. Implementations and canonical outputs
under `work/`. Nothing committed, nothing sent, nothing copied between arm
directories.