# PREREGISTRATION — Arm A2-id
**Study:** openhands-index recon/discovery/lawful run (HANDOFF v3, 2026-07-24)
**Arm:** A2-id — identified corpus, Executor A2 (Claude, fresh session; model id `claude-fable-5`)
**Date written:** 2026-07-25 (US/Eastern), before any reconciliation computation.
**Status at time of writing:** Recon R1–R5 executed; schema-only inspection of both
parquets executed (column names, dtypes, row counts, null counts, decimal-precision
census). **No cross-artifact join, aggregation, comparison, or reconciliation has been
computed at the time this file is written.**

---

## 0. What was inspected before this preregistration (disclosed per L1)

- Bytes fetched and pinned (sha256, sizes, HTTP status, `x-repo-commit`) for
  `test.parquet`, `instances.parquet`, `README.md` of `OpenHands/openhands-index`
  at main = `db6504dfbb2ac2c905c71f73f430bf9cfdc55018`.
- The dataset README (a publisher claim, to be tested, not assumed).
- Publisher pipeline code `scripts/publish_hf_dataset.py` at source commit
  `3015ac612e7196f428e6e8a3948965d32d9a3331` of
  `github.com/OpenHands/openhands-index-results` (R3 surface; its statements are
  claims entering the policy enumeration below, not privileged truths).
- Schemas: `test.parquet` = 34 rows × 38 columns; `instances.parquet` = 40,643 rows ×
  9 columns (`id, agent_name, agent_type, language_model, benchmark, category,
  instance_id, resolved (bool, 475 nulls), cost (double, 436 nulls)`).
- Decimal-precision census of `test.parquet` float columns (formatting property):
  per-category `*_score` max 1 decimal; `average_score` max 2; all `*_cost` max 4;
  `average_runtime` max 2; per-category `*_runtime` max 0–1.
- The workspace arrived pre-staged by the custodian with raw public fetches (parquets,
  HTTP headers, HF/GitHub API captures, Space code). I verified the parquet bytes by
  independent re-fetch (hashes identical). No prior-analysis material was encountered.

No value of any `test.parquet` cell has been compared to any recomputation at this
point. The precision census and null counts above are the only value-derived facts
consumed pre-registration, and they are used solely to set tolerance units (P6).

## 1. Designated recompute target (P4/P12)

**Target artifact: `test.parquet` (config `default`), all 34 rows × 38 columns =
1,292 cells.** Source artifact for recomputation: `instances.parquet`. Direction:
recompute `test.parquet` values from `instances.parquet` alone (no network, no other
inputs in the recompute path — O9).

## 2. Correspondence derivation (P4)

### 2.1 Row mapping (join key) — derived by search, not assumed
Candidate join keys, all to be evaluated; winner = highest (coverage × uniqueness);
runner-up reported:
- K1: `test.id` = `instances.id`
- K2: `test.language_model` = `instances.language_model`
- K3: (`test.agent_name`, `test.language_model`) = same pair in instances
- K4: case-insensitive variant of K1

Coverage = fraction of test rows with ≥1 instance row under the key; uniqueness =
no test row maps to instance rows that also map to another test row. Both directions
reported (test rows without instance groups; instance groups without test rows).

### 2.2 Column-family mapping (benchmark → per-category columns)
Candidate mappings from `instances.benchmark` to test column prefixes:
- C1: the (benchmark, category) pairing observed inside `instances.parquet` itself,
  slugged `category.lower().replace(' ', '_')`.
- C2: the README table pairing (claim).
- C3: the publisher-code constant `BENCHMARK_TO_CATEGORY` (claim).
All three evaluated for consistency; disagreements reported. If C1 is internally
inconsistent (one benchmark ↔ multiple categories), that is reported and C1 decides
per majority with the tie reported.

### 2.3 Column correspondences and no-antecedent declarations
Addressed columns (candidate antecedent exists in `instances.parquet`):
- Identity/categorical: `id` (join key itself — classified structural if used as key),
  `agent_name`, `agent_type`, `language_model` — antecedent = the set of values in the
  model's instance rows.
- `<cat>_score` ×5 — antecedent = `resolved` over the model×benchmark rows.
- `<cat>_cost` ×5 — antecedent = `cost` over the model×benchmark rows.
- `average_score`, `average_cost` — antecedent = compositions of the above.
- `categories_completed` — antecedent = benchmark presence per model.

No-antecedent declarations (no column or combination in `instances.parquet` carries
the quantity; reason in parentheses):
- `sdk_version`, `openness`, `country`, `supports_vision`, `release_date`
  (metadata.json provenance per publisher code; nothing in instances schema).
- `average_runtime`, `<cat>_runtime` ×5 (no runtime column in instances).
- `<cat>_logs_url` ×5, `<cat>_visualization_url` ×5 (no URL columns in instances).
Total no-antecedent columns: 21; addressed columns: 17.

## 3. Policy enumeration (P5) — declared exhaustively, all reported

Notation: for a model m and benchmark b, G(m,b) = instance rows joined under the
winning key and column mapping. R⁺ = count(resolved == True), N_all = |G|,
N_obs = count(resolved non-null), N_cost = count(cost non-null).

### 3.1 Per-category `<cat>_score` (5 columns)
Aggregation family: proportion of resolved, scaled.
- Denominator policies: **D_all** = N_all; **D_obs** = N_obs.
- Missing-value policies (on `resolved`): **M_false** (null → not resolved; pairs with
  D_all) and **M_excl** (null excluded; pairs with D_obs). The pairing is forced by
  coherence; both pairs are computed: S1 = 100·R⁺/N_all, S2 = 100·R⁺/N_obs.
- Scale candidates: percent (×100) and fraction (×1) — both computed.
- Rounding candidates: none; round-half-even to 1 dp (published precision);
  round-half-even to 4 dp (publisher-code constant).
- Conditioned candidates (P5 conditional clause): none beyond the b-conditioning that
  defines the column family; no further structure derived under P4 admits more.
**Primary policy (for P9 nulls and narrative ordering only; all policies reported):
S1_pct** = 100·R⁺/N_all, unrounded comparison plus round-to-1dp comparison.

### 3.2 Per-category `<cat>_cost` (5 columns)
- Aggregation candidates: **mean**, **median**, **sum/N_all**, **sum/N_resolved⁺**
  (sum of non-null costs over count of resolved==True).
- Missing-value policies (on `cost`): **M_excl** (non-null only), **M_zero**
  (null → 0.0, denominator N_all).
- Denominator policies: N_cost vs N_all (these instantiate the two missing policies
  for the mean; for median, M_excl only — a null-as-zero median is also computed for
  completeness).
- Rounding: none; round-half-even to 4 dp.
**Primary: mean over non-null costs (M_excl), unrounded + 4dp.**

### 3.3 `average_score`
- **AS-macro-raw**: mean of the 5 per-benchmark recomputed scores (primary per-cat
  policy), then round-half-even to 4 dp (publisher-code form) and to 2 dp.
- **AS-macro-pub**: mean of the 5 per-benchmark scores first rounded to 1 dp.
- **AS-micro**: 100·ΣR⁺/ΣN over all the model's rows (both D_all and D_obs).
- Denominator variants: over benchmarks present (categories completed) vs over all 5
  (missing benchmark → excluded vs → 0); both computed.
**Primary: AS-macro-raw over benchmarks present, D_all.**

### 3.4 `average_cost`
- **AC-macro-raw**: mean of the 5 recomputed per-benchmark mean costs (M_excl).
- **AC-macro-pub**: same with per-benchmark values rounded to 4 dp first.
- **AC-micro**: mean over all the model's non-null costs; plus M_zero variant.
- Denominator variants: benchmarks present vs all 5; both computed.
**Primary: AC-macro-raw over benchmarks present.**

### 3.5 `categories_completed`
- **CC1** = count of distinct benchmarks with ≥1 instance row for the model.
- **CC2** = count of distinct benchmarks with ≥1 non-null `resolved` row.
Integer comparison, exact only.
**Primary: CC1.**

### 3.6 Categorical columns (`agent_name`, `agent_type`, `language_model`)
- **U-unique**: reconciles iff the value set over the model's instance rows is a
  singleton equal to the test value.
- **U-major**: reconciles iff the modal value equals the test value; multiplicity of
  the value set reported.
**Primary: U-unique.**

No single policy fits every column (permitted statement under P5): counts and
categoricals admit no denominator/missing grid; this is stated rather than forced.

## 4. Tolerance tiers (P6) — declared now, in published units, not adjusted later

Score columns (percent points): **T0** = equality after rounding recomputed value
half-even to the column's published precision (1 dp per-category, 2 dp average);
**T1** = |Δ| ≤ 0.05; **T2** = |Δ| ≤ 0.5; else **divergent** (under every declared
policy → cell class "divergent"). Δ measured on unrounded recompute vs published.
Cost columns (USD): **T0** = equality at 4 dp; **T1** = |Δ| ≤ 0.005; **T2** =
|Δ| ≤ 0.05; else divergent.
`categories_completed`: exact integer equality only (T0 or divergent).
Categorical: equality or divergent.
Per-column and per-row deviations reported, plus count of rows matching at each
column's own published precision (P6), for every declared policy, not only primary.

## 5. Independent implementations (P7) and determinism (P8)

- **Impl A**: Python + pandas (+pyarrow parquet reader via pandas).
- **Impl B**: Python + pyarrow only for byte reading, all aggregation in pure-Python
  dict/loop code; no pandas import; written after A without referring to A's code
  beyond this preregistration. Shared inputs: the two parquets and this file only.
- **Agreement threshold (declared):** for every (row, column, policy) recomputed
  value: both null, or |a−b| ≤ 1e-9 absolute; every cell classification identical;
  every match count identical. Anything less fails the P7 gate and is reported.
- **P8:** each implementation executed as two separate OS process invocations;
  canonical output files (sorted keys, fixed float repr `repr(float)`, `\n` EOL)
  byte-compared with `cmp`; diff results recorded in RESULTS.

## 6. Negative control (P9) — permutation null

- Scheme: permute the assignment of the 34 join-key groups of `instances.parquet` to
  the 34 `test.parquet` rows (uniform random permutation of group labels).
- Seed: `numpy.random.default_rng(20260725)`; draws: **1000**; identical scheme in
  both implementations is NOT required (P9 lives in the RESULTS transcript from
  Impl A; Impl B reproduces the observed statistics, and the null is additionally
  re-run in Impl B with the same seed via numpy — numpy is permitted in Impl B for
  the null only, since the null is not a reconciliation of the corpus).
- Statistics per addressed column, under its primary policy, per draw:
  (a) count of rows reconciled at T0; (b) mean absolute deviation (numeric columns);
  (c) match count (categorical/integer columns).
- Reported per column: **median and 5th percentile** of the null distribution of each
  statistic (5th percentile of MAD is the null's favorable tail; for match counts the
  95th percentile is also recorded so the favorable tail is visible in both
  directions), plus the observed value alongside.
- Where a column's null distribution is degenerate (zero-variance column), that is a
  structural floor: stated as structural, ratios against it undefined, not infinite.

## 7. Spread (P10), rankings (L4), and integrity extras

- Every addressed column's own range (min, max) over the 34 rows is reported; a
  zero-variance column's match is flagged as carrying no discriminating information
  and the cells classified **structural**.
- Ranking: the census of ties in published `average_score` at its published precision
  is reported, and whether the published sort order (descending `average_score`) is
  decidable; the same census for the primary recomputed average. No statement about
  which records produced any movement (L4).
- **M-H (hash reconstruction, declared measurement):** recompute the publisher-code
  `content_hash` (sha256 over both tables serialized as column-sorted CSV,
  `float_format="%.10g"`, joined by `\n--\n`) from the shipped parquets and compare to
  the shipped `.source_hash` (`886a3ebabd70879b712deb36fcfc190199e9c91a0ea538a89b156a20663fda29`).
  Equal / not equal is reported either way; this tests serialization identity of the
  shipped bytes against the publisher's own no-op detector, nothing more.

## 8. Cell accounting (P12) — classification procedure

Each of the 1,292 cells receives exactly one class, by this precedence order,
declared now:
1. **no antecedent** — cell in a no-antecedent column (§2.3), or in an addressed
   column where the join yields zero instance rows for that model (per-category
   columns: zero rows for that model×benchmark).
2. **structural** — cell in an addressed column whose `test.parquet` values have zero
   variance across the 34 rows (match carries no discriminating information), or
   whose value is schema-forced.
3. **reconciled at tier N under policy P** — the best (lowest-N) tier achieved by any
   declared policy; the achieving policy named; all policies' tiers retained in the
   transcript.
4. **divergent under every declared policy** — otherwise.
Counts by class must sum to 1,292; the sum is printed in RESULTS.

## 9. Row-treatment rule (§3.5) — declared before computing

Rows receiving individual mention in the report are exactly:
(a) per addressed numeric column, the row(s) of maximum |Δ| under the primary policy,
    ties included, named mechanically;
(b) the executor-corresponding rows under §10 below (both executors, symmetric);
(c) rows involved in join failures (no instance group), if any.
No other row is discussed individually. No characterization of what produced a
residual (L2/L4).

## 10. Self-identification rule (§3.1) — declared before looking

The executing model of this arm is `claude-fable-5` (system-reported model id).
**Mapping rule:** a `test.parquet` row corresponds to an executor iff
`lower(language_model)` equals the executor's model id lowercased; if no exact
match, the fallback is substring match on the model family token (`fable` for A2).
Applied symmetrically to Executor B2 (Gemini CLI): B2's pinned model id is **unknown
to this arm** (recorded fact); therefore B2's candidate set = rows where
`lower(language_model)` begins with `gemini`, reported as a set with the note that
the pin is unknown, not resolved by choosing (P11).
Own-row values, rank, and the rule above are reported per §3.1. If the §9(a) rule
lands on an executor row, that is recorded as the §3.3 self-example-hazard outcome.

## 11. What locks this arm

This preregistration, then: PREDRAFT (both directions) → computation → RESULTS →
REPORT (containing this file verbatim, boundary section, "what this does not show").
Lock = those files on disk in `habitat_arms/iter2/A2-id/`. Envelope 2′ is requested
from the custodian only after lock. No sends, commits, pushes, or contact (L8).

## 12. Additions log (L1)

(Any addition is timestamped, withdraws nothing, and precedes the measurement it
governs.)

- **A1, 2026-07-25, before any computation ran.** For tie-set determination in §9(a)
  and for cross-implementation comparison of max-|Δ| row sets, |Δ| values are rounded
  half-even to 9 decimal places before argmax/tie grouping. Reason: makes tie sets
  well-defined across two implementations whose floating-point summation order may
  differ within the declared 1e-9 agreement threshold.
- **A2, 2026-07-25, before any computation ran.** The M-H hash reconstruction (§7)
  executes in Implementation A only. Reason: the object under test is the publisher's
  pandas CSV serialization; a pure-Python reimplementation of that serializer would
  test my CSV writer, not the publisher's. Single-implementation status is disclosed
  in RESULTS and M-H is excluded from the P7 gate.
- **A3, 2026-07-25, before any computation ran.** A cell whose antecedent group exists
  but whose every declared policy yields an undefined recomputation (no non-null
  antecedent values) is classified **no antecedent**, with the distinct reason
  "antecedent rows present, all values null" recorded in the transcript. Reason:
  "divergent under every declared policy" would otherwise mislabel cells where no
  policy produced a number at all.