================================================================================ RESULTS — Arm A2-id — 2026-07-25 Study: openhands-index recon/discovery/lawful run (HANDOFF v3) This transcript contains: both implementations × both process invocations, byte-diffs (P8), the P7 cross-implementation gate, per-column permutation nulls (P9, seed 20260725, 1000 draws), per-column policy tables (P5/P6), per-row deviations under primary policies (P6), tie censuses (L4), cell accounting (P12), self-identification rows (§3.1), and the M-H hash reconstruction. ENVIRONMENT Shared MacBook Air (Mac16,13, Apple M4), macOS 26.2 (25C56), arm64, host MacBook-Air.local. Shared machine; NO machine independence (§3.6). conda env "habitat" at /Applications/anaconda3: Python 3.11.13, numpy 1.26.4, pandas 2.3.2, pyarrow 14.0.2. INPUTS (P2 pins; fetched from HF main = db6504dfbb2ac2c905c71f73f430bf9cfdc55018, HTTP 302->200 via CDN; staged custodian copies byte-identical to my fresh fetch) test.parquet 44,323 bytes sha256 966544ad62ba160fec79c73bdd5295df521743c733f9fbcca1ed0f82833d827a (matches Appendix A) instances.parquet 435,757 bytes sha256 f456e937771bdd45815cacd6458433e0e750be0a2a6bcd5daf91670b151968a5 (matches Appendix A) README.md 5,226 bytes sha256 4e134a2fa3ca64cb93216a008f74a7bb67a513f73f4755c7993a4052067b60fb shipped .source_hash = 886a3ebabd70879b712deb36fcfc190199e9c91a0ea538a89b156a20663fda29 RECOMPUTE PATH: local bytes only; no network (O9). Preregistration: PREREG_A2-id_2026-07-25.md (written before any computation below). ================================================================================ SECTION 1 — RUN TRANSCRIPT (P7 implementations, P8 process repeats, byte-diffs, P7 gate) === RUN LOG 2026-07-25 22:54:02 UTC === --- impl_A run 1 --- impl_A wrote out_A_run1.json; cell totals: {'reconciled': 340, 'divergent': 102, 'no_antecedent': 714, 'structural': 136, 'total': 1292, 'reconciled_by_tier': {'T0': 199, 'T1': 71, 'T2': 70}} --- impl_A run 2 (separate process) --- impl_A wrote out_A_run2.json; cell totals: {'reconciled': 340, 'divergent': 102, 'no_antecedent': 714, 'structural': 136, 'total': 1292, 'reconciled_by_tier': {'T0': 199, 'T1': 71, 'T2': 70}} --- impl_B run 1 --- impl_B wrote out_B_run1.json; cell totals: {'reconciled': 340, 'divergent': 102, 'no_antecedent': 714, 'structural': 136, 'total': 1292, 'reconciled_by_tier': {'T0': 199, 'T1': 71, 'T2': 70}} --- impl_B run 2 (separate process) --- impl_B wrote out_B_run2.json; cell totals: {'reconciled': 340, 'divergent': 102, 'no_antecedent': 714, 'structural': 136, 'total': 1292, 'reconciled_by_tier': {'T0': 199, 'T1': 71, 'T2': 70}} --- P8 byte-diff impl_A run1 vs run2 --- IDENTICAL (byte-for-byte) --- P8 byte-diff impl_B run1 vs run2 --- IDENTICAL (byte-for-byte) 6f6ebf6aa853bc5ebd21587c26f2770776adf210e4528224d2939cbb5000f7bc out_A_run1.json 6f6ebf6aa853bc5ebd21587c26f2770776adf210e4528224d2939cbb5000f7bc out_A_run2.json 4840a810430e9b3375c9e87ed81215e207f5c5ce76d3acf9aed1fe4c6b0dd243 out_B_run1.json 4840a810430e9b3375c9e87ed81215e207f5c5ce76d3acf9aed1fe4c6b0dd243 out_B_run2.json --- P7 comparator impl_A vs impl_B --- P7 comparator: out_A_run1.json vs out_B_run1.json compared sections: ['cells', 'colmap', 'columns', 'instances_side', 'join', 'nulls', 'self_id', 'tie_census'] mismatches: 0 P7 GATE: PASS exit=0 SECTION 2 — JOIN / COLUMN MAPPING / INSTANCES-SIDE / HASH CHECK / TIE CENSUS / CELL TOTALS / PER-COLUMN POLICY TABLES / PERMUTATION NULLS (P9 — per addressed column; observed alongside null median, 5th and 95th percentiles) == JOIN == { "candidates": { "K1_id": { "coverage": 1.0, "n_instance_groups": 34, "test_key_unique": true }, "K2_language_model": { "coverage": 1.0, "n_instance_groups": 34, "test_key_unique": true }, "K3_agentname_lm": { "coverage": 1.0, "n_instance_groups": 34, "test_key_unique": true }, "K4_id_lower": { "coverage": 1.0, "n_instance_groups": 34, "test_key_unique": true } }, "runner_up": "K2_language_model", "winner": "K1_id" } == COLMAP == { "C1_ambiguous_benchmarks": [], "C1_pairs": [ [ "commit0", "Greenfield" ], [ "gaia", "Information Gathering" ], [ "swe-bench", "Issue Resolution" ], [ "swe-bench-multimodal", "Frontend" ], [ "swt-bench", "Testing" ] ], "agrees_with_C2_README_and_C3_code": { "commit0": true, "gaia": true, "swe-bench": true, "swe-bench-multimodal": true, "swt-bench": true }, "mapping_used": { "commit0": "greenfield", "gaia": "information_gathering", "swe-bench": "issue_resolution", "swe-bench-multimodal": "frontend", "swt-bench": "testing" } } == INSTANCES SIDE == { "cost_nulls": 436, "distinct_benchmarks": [ "commit0", "gaia", "swe-bench", "swe-bench-multimodal", "swt-bench" ], "groups_without_test_row": [], "n_groups": 34, "n_rows": 40643, "resolved_nulls": 475, "test_rows_without_group": [] } == HASH CHECK == { "equal": true, "executed": true, "recomputed": "886a3ebabd70879b712deb36fcfc190199e9c91a0ea538a89b156a20663fda29", "shipped_source_hash": "886a3ebabd70879b712deb36fcfc190199e9c91a0ea538a89b156a20663fda29" } == TIE CENSUS == { "published_average_score": { "adjacent_ties_in_desc_sort": 0, "distinct": 34, "n": 34, "strictly_decidable": true, "tied_pairs": 0 }, "published_sort_descending": true, "recomputed_primary_2dp": { "adjacent_ties_in_desc_sort": 0, "distinct": 34, "n": 34, "strictly_decidable": true, "tied_pairs": 0 }, "recomputed_primary_raw": { "adjacent_ties_in_desc_sort": 0, "distinct": 34, "n": 34, "strictly_decidable": true, "tied_pairs": 0 } } == CELL TOTALS == { "divergent": 102, "no_antecedent": 714, "reconciled": 340, "reconciled_by_tier": { "T0": 199, "T1": 71, "T2": 70 }, "structural": 136, "total": 1292 } == PER-COLUMN == agent_name fam=categorical zv=True range=None cells={'divergent': 0, 'no_antecedent': 0, 'reconciled': 0, 'structural': 34} U_major: t0=34 t1=0 t2=0 div=0 undef=0 U_unique: t0=34 t1=0 t2=0 div=0 undef=0 primary=U_unique max|d|=None rows=[] agent_type fam=categorical zv=True range=None cells={'divergent': 0, 'no_antecedent': 0, 'reconciled': 0, 'structural': 34} U_major: t0=34 t1=0 t2=0 div=0 undef=0 U_unique: t0=34 t1=0 t2=0 div=0 undef=0 primary=U_unique max|d|=None rows=[] average_cost fam=cost zv=False range=[0.1533, 5.538] cells={'divergent': 17, 'no_antecedent': 0, 'reconciled': 17, 'structural': 0} AC_macro_4dp: t0=4 t1=4 t2=9 div=17 undef=0 AC_macro_all5_zero: t0=4 t1=4 t2=9 div=17 undef=0 AC_macro_raw: t0=4 t1=4 t2=9 div=17 undef=0 AC_micro_excl: t0=0 t1=0 t2=1 div=33 undef=0 AC_micro_zero: t0=0 t1=0 t2=2 div=32 undef=0 primary=AC_macro_raw max|d|=0.594564538 rows=['OpenHands/Gemini-3.5-Flash'] average_score fam=score zv=False range=[15.48, 81.0] cells={'divergent': 21, 'no_antecedent': 0, 'reconciled': 13, 'structural': 0} AS_macro_all5_zero: t0=1 t1=6 t2=4 div=23 undef=0 AS_macro_pub1dp: t0=6 t1=0 t2=5 div=23 undef=0 AS_macro_raw_Dall: t0=1 t1=6 t2=4 div=23 undef=0 AS_macro_raw_Dobs: t0=1 t1=6 t2=6 div=21 undef=0 AS_micro_Dall: t0=0 t1=0 t2=0 div=34 undef=0 AS_micro_Dobs: t0=0 t1=0 t2=0 div=34 undef=0 primary=AS_macro_raw_Dall max|d|=2.752574296 rows=['OpenHands/Gemini-3.1-Pro'] categories_completed fam=count zv=True range=[5.0, 5.0] cells={'divergent': 0, 'no_antecedent': 0, 'reconciled': 0, 'structural': 34} CC1_any_row: t0=34 t1=0 t2=0 div=0 undef=0 CC2_any_obs: t0=34 t1=0 t2=0 div=0 undef=0 primary=CC1_any_row max|d|=0 rows=['OpenHands/DeepSeek-V3.2-Reasoner', 'OpenHands/DeepSeek-V4-Pro', 'OpenHands/GLM-4.7', 'OpenHands/GLM-5', 'OpenHands/GLM-5.1', 'OpenHands/GPT-5.2', 'OpenHands/GPT-5.2-Codex', 'OpenHands/GPT-5.4', 'OpenHands/GPT-5.5', 'OpenHands/Gemini-3-Flash', 'OpenHands/Gemini-3-Pro', 'OpenHands/Gemini-3.1-Pro', 'OpenHands/Gemini-3.5-Flash', 'OpenHands/Kimi-K2-Thinking', 'OpenHands/Kimi-K2.5', 'OpenHands/Kimi-K2.6', 'OpenHands/MiniMax-M2.1', 'OpenHands/MiniMax-M2.5', 'OpenHands/MiniMax-M2.7', 'OpenHands/MiniMax-M3', 'OpenHands/Nemotron-3-Nano', 'OpenHands/Nemotron-3-Super', 'OpenHands/Qwen3-Coder-480B', 'OpenHands/Qwen3-Coder-Next', 'OpenHands/Qwen3.5-Flash', 'OpenHands/Qwen3.6-Plus', 'OpenHands/Trinity-Large-Thinking', 'OpenHands/claude-fable-5', 'OpenHands/claude-opus-4-5', 'OpenHands/claude-opus-4-6', 'OpenHands/claude-opus-4-7', 'OpenHands/claude-opus-4-8', 'OpenHands/claude-sonnet-4-5', 'OpenHands/claude-sonnet-4-6'] frontend_cost fam=cost zv=False range=[0.19, 6.92] cells={'divergent': 18, 'no_antecedent': 0, 'reconciled': 16, 'structural': 0} C_mean_excl: t0=4 t1=4 t2=4 div=22 undef=0 C_mean_zero: t0=3 t1=5 t2=6 div=20 undef=0 C_median_excl: t0=0 t1=0 t2=3 div=31 undef=0 C_median_zero: t0=0 t1=0 t2=3 div=31 undef=0 C_sum_over_resolved: t0=0 t1=0 t2=0 div=34 undef=0 primary=C_mean_excl max|d|=3.308446717 rows=['OpenHands/GLM-5.1'] frontend_score fam=score zv=False range=[16.2, 70.6] cells={'divergent': 22, 'no_antecedent': 0, 'reconciled': 12, 'structural': 0} S1_frac_Dall: t0=0 t1=0 t2=0 div=34 undef=0 S1_pct_Dall: t0=12 t1=0 t2=0 div=22 undef=0 S2_frac_Dobs: t0=0 t1=0 t2=0 div=34 undef=0 S2_pct_Dobs: t0=10 t1=0 t2=2 div=22 undef=0 primary=S1_pct_Dall max|d|=13.707843137 rows=['OpenHands/Gemini-3.1-Pro'] greenfield_cost fam=cost zv=False range=[0.01, 12.49] cells={'divergent': 0, 'no_antecedent': 0, 'reconciled': 34, 'structural': 0} C_mean_excl: t0=6 t1=7 t2=2 div=19 undef=0 C_mean_zero: t0=3 t1=28 t2=0 div=3 undef=0 C_median_excl: t0=1 t1=1 t2=2 div=30 undef=0 C_median_zero: t0=1 t1=0 t2=6 div=27 undef=0 C_sum_over_resolved: t0=0 t1=0 t2=0 div=33 undef=1 primary=C_mean_excl max|d|=2.106382731 rows=['OpenHands/claude-opus-4-5'] greenfield_score fam=score zv=False range=[0.0, 62.5] cells={'divergent': 0, 'no_antecedent': 0, 'reconciled': 34, 'structural': 0} S1_frac_Dall: t0=1 t1=0 t2=0 div=33 undef=0 S1_pct_Dall: t0=34 t1=0 t2=0 div=0 undef=0 S2_frac_Dobs: t0=1 t1=0 t2=0 div=33 undef=0 S2_pct_Dobs: t0=34 t1=0 t2=0 div=0 undef=0 primary=S1_pct_Dall max|d|=0.05 rows=['OpenHands/GLM-5', 'OpenHands/GPT-5.2-Codex', 'OpenHands/GPT-5.4', 'OpenHands/GPT-5.5', 'OpenHands/Gemini-3-Flash', 'OpenHands/Kimi-K2.5', 'OpenHands/MiniMax-M2.1', 'OpenHands/MiniMax-M2.7', 'OpenHands/Nemotron-3-Nano', 'OpenHands/claude-opus-4-6', 'OpenHands/claude-opus-4-7'] information_gathering_cost fam=cost zv=False range=[0.04, 7.91] cells={'divergent': 5, 'no_antecedent': 0, 'reconciled': 29, 'structural': 0} C_mean_excl: t0=7 t1=5 t2=13 div=9 undef=0 C_mean_zero: t0=4 t1=9 t2=11 div=10 undef=0 C_median_excl: t0=0 t1=0 t2=4 div=30 undef=0 C_median_zero: t0=0 t1=0 t2=4 div=30 undef=0 C_sum_over_resolved: t0=0 t1=0 t2=2 div=32 undef=0 primary=C_mean_excl max|d|=0.280643806 rows=['OpenHands/Gemini-3.1-Pro'] information_gathering_score fam=score zv=False range=[8.5, 86.1] cells={'divergent': 2, 'no_antecedent': 0, 'reconciled': 32, 'structural': 0} S1_frac_Dall: t0=0 t1=0 t2=0 div=34 undef=0 S1_pct_Dall: t0=32 t1=0 t2=0 div=2 undef=0 S2_frac_Dobs: t0=0 t1=0 t2=0 div=34 undef=0 S2_pct_Dobs: t0=32 t1=0 t2=0 div=2 undef=0 primary=S1_pct_Dall max|d|=5.5 rows=['OpenHands/Nemotron-3-Nano'] issue_resolution_cost fam=cost zv=False range=[0.0395, 2.1639] cells={'divergent': 2, 'no_antecedent': 0, 'reconciled': 32, 'structural': 0} C_mean_excl: t0=6 t1=8 t2=18 div=2 undef=0 C_mean_zero: t0=5 t1=8 t2=19 div=2 undef=0 C_median_excl: t0=0 t1=0 t2=2 div=32 undef=0 C_median_zero: t0=0 t1=0 t2=2 div=32 undef=0 C_sum_over_resolved: t0=0 t1=0 t2=2 div=32 undef=0 primary=C_mean_excl max|d|=0.134640434 rows=['OpenHands/Kimi-K2-Thinking'] issue_resolution_score fam=score zv=False range=[34.2, 95.8] cells={'divergent': 0, 'no_antecedent': 0, 'reconciled': 34, 'structural': 0} S1_frac_Dall: t0=0 t1=0 t2=0 div=34 undef=0 S1_pct_Dall: t0=33 t1=0 t2=1 div=0 undef=0 S2_frac_Dobs: t0=0 t1=0 t2=0 div=34 undef=0 S2_pct_Dobs: t0=29 t1=0 t2=5 div=0 undef=0 primary=S1_pct_Dall max|d|=0.2 rows=['OpenHands/Gemini-3.1-Pro'] language_model fam=categorical zv=False range=None cells={'divergent': 0, 'no_antecedent': 0, 'reconciled': 34, 'structural': 0} U_major: t0=34 t1=0 t2=0 div=0 undef=0 U_unique: t0=34 t1=0 t2=0 div=0 undef=0 primary=U_unique max|d|=None rows=[] testing_cost fam=cost zv=False range=[0.0354, 2.085] cells={'divergent': 3, 'no_antecedent': 0, 'reconciled': 31, 'structural': 0} C_mean_excl: t0=6 t1=13 t2=12 div=3 undef=0 C_mean_zero: t0=5 t1=13 t2=13 div=3 undef=0 C_median_excl: t0=0 t1=1 t2=8 div=25 undef=0 C_median_zero: t0=0 t1=1 t2=8 div=25 undef=0 C_sum_over_resolved: t0=0 t1=0 t2=2 div=31 undef=1 primary=C_mean_excl max|d|=0.148855991 rows=['OpenHands/Kimi-K2-Thinking'] testing_score fam=score zv=False range=[7.6, 91.9] cells={'divergent': 12, 'no_antecedent': 0, 'reconciled': 22, 'structural': 0} S1_frac_Dall: t0=0 t1=0 t2=0 div=34 undef=0 S1_pct_Dall: t0=15 t1=0 t2=7 div=12 undef=0 S2_frac_Dobs: t0=0 t1=0 t2=0 div=34 undef=0 S2_pct_Dobs: t0=15 t1=0 t2=7 div=12 undef=0 primary=S1_pct_Dall max|d|=14.077245509 rows=['OpenHands/GLM-5'] == NULLS == agent_name {"note": "null distribution degenerate; floor structural, ratios undefined", "structural_zero_variance": true} agent_type {"note": "null distribution degenerate; floor structural, ratios undefined", "structural_zero_variance": true} average_cost {"mad_median": 1.2735428080158875, "mad_p5": 1.0654388442965514, "n_draws": 1000, "observed_mad": 0.14320260481652225, "observed_t0": 4, "seed": 20260725, "t0_median": 0.0, "t0_p5": 0.0, "t0_p95": 1.0} average_score {"mad_median": 14.87084154072382, "mad_p5": 12.431161272943912, "n_draws": 1000, "observed_mad": 1.23388068302469, "observed_t0": 1, "seed": 20260725, "t0_median": 0.0, "t0_p5": 0.0, "t0_p95": 0.0} categories_completed {"note": "null distribution degenerate; floor structural, ratios undefined", "structural_zero_variance": true} frontend_cost {"mad_median": 1.4033380821366443, "mad_p5": 1.1747099631044813, "n_draws": 1000, "observed_mad": 0.42171959509661194, "observed_t0": 4, "seed": 20260725, "t0_median": 0.0, "t0_p5": 0.0, "t0_p95": 1.0} frontend_score {"mad_median": 13.302934514794456, "mad_p5": 11.36521759638249, "n_draws": 1000, "observed_mad": 5.761275053697801, "observed_t0": 12, "seed": 20260725, "t0_median": 1.0, "t0_p5": 0.0, "t0_p95": 3.0} greenfield_cost {"mad_median": 3.6332918523573388, "mad_p5": 3.0273756710987274, "n_draws": 1000, "observed_mad": 0.4513671909637705, "observed_t0": 6, "seed": 20260725, "t0_median": 0.0, "t0_p5": 0.0, "t0_p95": 1.0} greenfield_score {"mad_median": 19.125, "mad_p5": 15.807352941176468, "n_draws": 1000, "observed_mad": 0.016176470588234976, "observed_t0": 34, "seed": 20260725, "t0_median": 5.0, "t0_p5": 2.0, "t0_p95": 8.0} information_gathering_cost {"mad_median": 0.8783791735777886, "mad_p5": 0.6930726342166862, "n_draws": 1000, "observed_mad": 0.03469798662041565, "observed_t0": 7, "seed": 20260725, "t0_median": 0.0, "t0_p5": 0.0, "t0_p95": 1.0} information_gathering_score {"mad_median": 24.224094632099096, "mad_p5": 19.62963830411336, "n_draws": 1000, "observed_mad": 0.28995915972833286, "observed_t0": 32, "seed": 20260725, "t0_median": 1.0, "t0_p5": 0.0, "t0_p95": 3.0} issue_resolution_cost {"mad_median": 0.6530343725176413, "mad_p5": 0.5379216007036475, "n_draws": 1000, "observed_mad": 0.017307726345861647, "observed_t0": 6, "seed": 20260725, "t0_median": 0.0, "t0_p5": 0.0, "t0_p95": 1.0} issue_resolution_score {"mad_median": 8.94705882352941, "mad_p5": 7.381764705882351, "n_draws": 1000, "observed_mad": 0.005882352941176554, "observed_t0": 33, "seed": 20260725, "t0_median": 1.0, "t0_p5": 0.0, "t0_p95": 4.0} language_model {"mad_median": null, "mad_p5": null, "n_draws": 1000, "observed_mad": null, "observed_t0": 34, "seed": 20260725, "t0_median": 1.0, "t0_p5": 0.0, "t0_p95": 3.0} testing_cost {"mad_median": 0.5581489595324347, "mad_p5": 0.4608154310215198, "n_draws": 1000, "observed_mad": 0.01627007491202649, "observed_t0": 6, "seed": 20260725, "t0_median": 0.0, "t0_p5": 0.0, "t0_p95": 1.0} testing_score {"mad_median": 19.239281272155146, "mad_p5": 15.853197193541973, "n_draws": 1000, "observed_mad": 1.3416681934848422, "observed_t0": 15, "seed": 20260725, "t0_median": 0.0, "t0_p5": 0.0, "t0_p95": 2.0} SECTION 3 — PER-ROW DEVIATIONS UNDER PRIMARY POLICIES (P6) PER-ROW DEVIATIONS UNDER PRIMARY POLICIES (recomputed, delta=rec-pub, tier; tier 0=T0 exact-at-published-precision, 1=T1, 2=T2, div=divergent under this policy) --- issue_resolution_score (primary=S1_pct_Dall, published range=[34.2, 95.8]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 71.600000 delta= 0.0 tier=0 OpenHands/DeepSeek-V4-Pro rec= 73.200000 delta= 0.0 tier=0 OpenHands/GLM-4.7 rec= 73.400000 delta= 0.0 tier=0 OpenHands/GLM-5 rec= 73.400000 delta= 0.0 tier=0 OpenHands/GLM-5.1 rec= 75.000000 delta= 0.0 tier=0 OpenHands/GPT-5.2 rec= 74.600000 delta= 0.0 tier=0 OpenHands/GPT-5.2-Codex rec= 73.800000 delta= 0.0 tier=0 OpenHands/GPT-5.4 rec= 75.600000 delta= 0.0 tier=0 OpenHands/GPT-5.5 rec= 78.200000 delta= 0.0 tier=0 OpenHands/Gemini-3-Flash rec= 74.600000 delta= 0.0 tier=0 OpenHands/Gemini-3-Pro rec= 70.600000 delta= 0.0 tier=0 OpenHands/Gemini-3.1-Pro rec= 77.000000 delta= 0.2 tier=2 OpenHands/Gemini-3.5-Flash rec= 78.600000 delta= 0.0 tier=0 OpenHands/Kimi-K2-Thinking rec= 69.200000 delta= 0.0 tier=0 OpenHands/Kimi-K2.5 rec= 68.800000 delta= 0.0 tier=0 OpenHands/Kimi-K2.6 rec= 74.600000 delta= 0.0 tier=0 OpenHands/MiniMax-M2.1 rec= 68.800000 delta= 0.0 tier=0 OpenHands/MiniMax-M2.5 rec= 72.600000 delta= 0.0 tier=0 OpenHands/MiniMax-M2.7 rec= 75.600000 delta= 0.0 tier=0 OpenHands/MiniMax-M3 rec= 76.400000 delta= 0.0 tier=0 OpenHands/Nemotron-3-Nano rec= 34.200000 delta= 0.0 tier=0 OpenHands/Nemotron-3-Super rec= 62.000000 delta= 0.0 tier=0 OpenHands/Qwen3-Coder-480B rec= 62.400000 delta= 0.0 tier=0 OpenHands/Qwen3-Coder-Next rec= 66.600000 delta= 0.0 tier=0 OpenHands/Qwen3.5-Flash rec= 62.000000 delta= 0.0 tier=0 OpenHands/Qwen3.6-Plus rec= 74.200000 delta= 0.0 tier=0 OpenHands/Trinity-Large-Thinking rec= 56.800000 delta= 0.0 tier=0 OpenHands/claude-fable-5 rec= 95.800000 delta= 0.0 tier=0 OpenHands/claude-opus-4-5 rec= 76.600000 delta= 0.0 tier=0 OpenHands/claude-opus-4-6 rec= 76.800000 delta= 0.0 tier=0 OpenHands/claude-opus-4-7 rec= 81.600000 delta= 0.0 tier=0 OpenHands/claude-opus-4-8 rec= 83.800000 delta= 0.0 tier=0 OpenHands/claude-sonnet-4-5 rec= 74.200000 delta= 0.0 tier=0 OpenHands/claude-sonnet-4-6 rec= 74.400000 delta= 0.0 tier=0 --- frontend_score (primary=S1_pct_Dall, published range=[16.2, 70.6]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 18.627451 delta= -9.27254902 tier=div OpenHands/DeepSeek-V4-Pro rec= 36.764706 delta= -0.035294118 tier=0 OpenHands/GLM-4.7 rec= 14.705882 delta= -7.394117647 tier=div OpenHands/GLM-5 rec= 26.470588 delta= -8.829411765 tier=div OpenHands/GLM-5.1 rec= 41.176471 delta= -0.023529412 tier=0 OpenHands/GPT-5.2 rec= 20.588235 delta= -10.311764706 tier=div OpenHands/GPT-5.2-Codex rec= 22.549020 delta= -13.350980392 tier=div OpenHands/GPT-5.4 rec= 36.764706 delta= -0.035294118 tier=0 OpenHands/GPT-5.5 rec= 38.235294 delta= 0.035294118 tier=0 OpenHands/Gemini-3-Flash rec= 14.705882 delta= -7.394117647 tier=div OpenHands/Gemini-3-Pro rec= 24.752475 delta= -12.047524752 tier=div OpenHands/Gemini-3.1-Pro rec= 30.392157 delta= -13.707843137 tier=div OpenHands/Gemini-3.5-Flash rec= 40.298507 delta= 0.598507463 tier=div OpenHands/Kimi-K2-Thinking rec= 21.568627 delta= -10.831372549 tier=div OpenHands/Kimi-K2.5 rec= 21.359223 delta= -11.440776699 tier=div OpenHands/Kimi-K2.6 rec= 41.176471 delta= -0.023529412 tier=0 OpenHands/MiniMax-M2.1 rec= 10.784314 delta= -5.415686275 tier=div OpenHands/MiniMax-M2.5 rec= 16.666667 delta= -8.333333333 tier=div OpenHands/MiniMax-M2.7 rec= 19.607843 delta= -8.292156863 tier=div OpenHands/MiniMax-M3 rec= 37.313433 delta= 0.513432836 tier=div OpenHands/Nemotron-3-Nano rec= 15.686275 delta= -5.21372549 tier=div OpenHands/Nemotron-3-Super rec= 20.588235 delta= -0.011764706 tier=0 OpenHands/Qwen3-Coder-480B rec= 17.647059 delta= -5.852941176 tier=div OpenHands/Qwen3-Coder-Next rec= 21.568627 delta= -9.331372549 tier=div OpenHands/Qwen3.5-Flash rec= 27.941176 delta= 0.041176471 tier=0 OpenHands/Qwen3.6-Plus rec= 30.882353 delta= -0.017647059 tier=0 OpenHands/Trinity-Large-Thinking rec= 25.000000 delta= 0.0 tier=0 OpenHands/claude-fable-5 rec= 70.588235 delta= -0.011764706 tier=0 OpenHands/claude-opus-4-5 rec= 27.722772 delta= -13.477227723 tier=div OpenHands/claude-opus-4-6 rec= 28.431373 delta= -13.368627451 tier=div OpenHands/claude-opus-4-7 rec= 48.529412 delta= 0.029411765 tier=0 OpenHands/claude-opus-4-8 rec= 50.000000 delta= 0.0 tier=0 OpenHands/claude-sonnet-4-5 rec= 25.490196 delta= -11.309803922 tier=div OpenHands/claude-sonnet-4-6 rec= 21.568627 delta= -9.331372549 tier=div --- greenfield_score (primary=S1_pct_Dall, published range=[0.0, 62.5]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 25.000000 delta= 0.0 tier=0 OpenHands/DeepSeek-V4-Pro rec= 12.500000 delta= 0.0 tier=0 OpenHands/GLM-4.7 rec= 12.500000 delta= 0.0 tier=0 OpenHands/GLM-5 rec= 31.250000 delta= 0.05 tier=0 OpenHands/GLM-5.1 rec= 37.500000 delta= 0.0 tier=0 OpenHands/GPT-5.2 rec= 50.000000 delta= 0.0 tier=0 OpenHands/GPT-5.2-Codex rec= 43.750000 delta= -0.05 tier=0 OpenHands/GPT-5.4 rec= 56.250000 delta= 0.05 tier=0 OpenHands/GPT-5.5 rec= 43.750000 delta= -0.05 tier=0 OpenHands/Gemini-3-Flash rec= 18.750000 delta= -0.05 tier=0 OpenHands/Gemini-3-Pro rec= 25.000000 delta= 0.0 tier=0 OpenHands/Gemini-3.1-Pro rec= 25.000000 delta= 0.0 tier=0 OpenHands/Gemini-3.5-Flash rec= 37.500000 delta= 0.0 tier=0 OpenHands/Kimi-K2-Thinking rec= 12.500000 delta= 0.0 tier=0 OpenHands/Kimi-K2.5 rec= 18.750000 delta= -0.05 tier=0 OpenHands/Kimi-K2.6 rec= 25.000000 delta= 0.0 tier=0 OpenHands/MiniMax-M2.1 rec= 18.750000 delta= -0.05 tier=0 OpenHands/MiniMax-M2.5 rec= 12.500000 delta= 0.0 tier=0 OpenHands/MiniMax-M2.7 rec= 18.750000 delta= -0.05 tier=0 OpenHands/MiniMax-M3 rec= 25.000000 delta= 0.0 tier=0 OpenHands/Nemotron-3-Nano rec= 6.250000 delta= 0.05 tier=0 OpenHands/Nemotron-3-Super rec= 12.500000 delta= 0.0 tier=0 OpenHands/Qwen3-Coder-480B rec= 0.000000 delta= 0.0 tier=0 OpenHands/Qwen3-Coder-Next rec= 25.000000 delta= 0.0 tier=0 OpenHands/Qwen3.5-Flash rec= 12.500000 delta= 0.0 tier=0 OpenHands/Qwen3.6-Plus rec= 25.000000 delta= 0.0 tier=0 OpenHands/Trinity-Large-Thinking rec= 12.500000 delta= 0.0 tier=0 OpenHands/claude-fable-5 rec= 62.500000 delta= 0.0 tier=0 OpenHands/claude-opus-4-5 rec= 37.500000 delta= 0.0 tier=0 OpenHands/claude-opus-4-6 rec= 56.250000 delta= 0.05 tier=0 OpenHands/claude-opus-4-7 rec= 56.250000 delta= 0.05 tier=0 OpenHands/claude-opus-4-8 rec= 62.500000 delta= 0.0 tier=0 OpenHands/claude-sonnet-4-5 rec= 12.500000 delta= 0.0 tier=0 OpenHands/claude-sonnet-4-6 rec= 50.000000 delta= 0.0 tier=0 --- testing_score (primary=S1_pct_Dall, published range=[7.6, 91.9]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 54.460094 delta= 0.860093897 tier=div OpenHands/DeepSeek-V4-Pro rec= 60.969977 delta= -7.130023095 tier=div OpenHands/GLM-4.7 rec= 50.710900 delta= 1.310900474 tier=div OpenHands/GLM-5 rec= 61.377246 delta= 14.077245509 tier=div OpenHands/GLM-5.1 rec= 69.745958 delta= -0.45404157 tier=2 OpenHands/GPT-5.2 rec= 74.238876 delta= 1.038875878 tier=div OpenHands/GPT-5.2-Codex rec= 66.974596 delta= -0.025404157 tier=0 OpenHands/GPT-5.4 rec= 70.438799 delta= 0.038799076 tier=0 OpenHands/GPT-5.5 rec= 83.371824 delta= -0.02817552 tier=0 OpenHands/Gemini-3-Flash rec= 70.997680 delta= 0.297679814 tier=2 OpenHands/Gemini-3-Pro rec= 68.750000 delta= 0.15 tier=2 OpenHands/Gemini-3.1-Pro rec= 74.826790 delta= -0.273210162 tier=2 OpenHands/Gemini-3.5-Flash rec= 76.388889 delta= 0.188888889 tier=2 OpenHands/Kimi-K2-Thinking rec= 47.344111 delta= 0.044110855 tier=0 OpenHands/Kimi-K2.5 rec= 62.037037 delta= 0.137037037 tier=2 OpenHands/Kimi-K2.6 rec= 70.438799 delta= 0.038799076 tier=0 OpenHands/MiniMax-M2.1 rec= 61.431871 delta= 0.03187067 tier=0 OpenHands/MiniMax-M2.5 rec= 68.129330 delta= 0.029330254 tier=0 OpenHands/MiniMax-M2.7 rec= 69.053118 delta= -0.046882217 tier=0 OpenHands/MiniMax-M3 rec= 81.062356 delta= -0.037644342 tier=0 OpenHands/Nemotron-3-Nano rec= 0.000000 delta= -7.6 tier=div OpenHands/Nemotron-3-Super rec= 42.032333 delta= -3.667667436 tier=div OpenHands/Qwen3-Coder-480B rec= 36.385542 delta= 1.485542169 tier=div OpenHands/Qwen3-Coder-Next rec= 42.494226 delta= -3.205773672 tier=div OpenHands/Qwen3.5-Flash rec= 36.951501 delta= -1.348498845 tier=div OpenHands/Qwen3.6-Plus rec= 60.969977 delta= -1.130023095 tier=div OpenHands/Trinity-Large-Thinking rec= 33.025404 delta= -0.274595843 tier=2 OpenHands/claude-fable-5 rec= 91.916859 delta= 0.016859122 tier=0 OpenHands/claude-opus-4-5 rec= 78.521940 delta= 0.021939954 tier=0 OpenHands/claude-opus-4-6 rec= 78.752887 delta= -0.047113164 tier=0 OpenHands/claude-opus-4-7 rec= 80.831409 delta= 0.031408776 tier=0 OpenHands/claude-opus-4-8 rec= 84.295612 delta= -0.004387991 tier=0 OpenHands/claude-sonnet-4-5 rec= 69.302326 delta= 0.502325581 tier=div OpenHands/claude-sonnet-4-6 rec= 54.041570 delta= 0.041570439 tier=0 --- information_gathering_score (primary=S1_pct_Dall, published range=[8.5, 86.1]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 50.303030 delta= 0.003030303 tier=0 OpenHands/DeepSeek-V4-Pro rec= 12.727273 delta= 0.027272727 tier=0 OpenHands/GLM-4.7 rec= 53.939394 delta= 0.039393939 tier=0 OpenHands/GLM-5 rec= 60.000000 delta= 0.0 tier=0 OpenHands/GLM-5.1 rec= 67.272727 delta= -0.027272727 tier=0 OpenHands/GPT-5.2 rec= 65.454545 delta= -0.045454545 tier=0 OpenHands/GPT-5.2-Codex rec= 70.909091 delta= 0.009090909 tier=0 OpenHands/GPT-5.4 rec= 82.424242 delta= 0.024242424 tier=0 OpenHands/GPT-5.5 rec= 86.060606 delta= -0.039393939 tier=0 OpenHands/Gemini-3-Flash rec= 58.787879 delta= -0.012121212 tier=0 OpenHands/Gemini-3-Pro rec= 44.242424 delta= 0.042424242 tier=0 OpenHands/Gemini-3.1-Pro rec= 81.818182 delta= 0.018181818 tier=0 OpenHands/Gemini-3.5-Flash rec= 84.810127 delta= 3.610126582 tier=div OpenHands/Kimi-K2-Thinking rec= 43.636364 delta= 0.036363636 tier=0 OpenHands/Kimi-K2.5 rec= 63.636364 delta= 0.036363636 tier=0 OpenHands/Kimi-K2.6 rec= 74.545455 delta= 0.045454545 tier=0 OpenHands/MiniMax-M2.1 rec= 40.606061 delta= 0.006060606 tier=0 OpenHands/MiniMax-M2.5 rec= 47.878788 delta= -0.021212121 tier=0 OpenHands/MiniMax-M2.7 rec= 25.454545 delta= -0.045454545 tier=0 OpenHands/MiniMax-M3 rec= 66.666667 delta= -0.033333333 tier=0 OpenHands/Nemotron-3-Nano rec= 14.000000 delta= 5.5 tier=div OpenHands/Nemotron-3-Super rec= 40.000000 delta= 0.0 tier=0 OpenHands/Qwen3-Coder-480B rec= 33.939394 delta= 0.039393939 tier=0 OpenHands/Qwen3-Coder-Next rec= 50.909091 delta= 0.009090909 tier=0 OpenHands/Qwen3.5-Flash rec= 49.696970 delta= -0.003030303 tier=0 OpenHands/Qwen3.6-Plus rec= 72.121212 delta= 0.021212121 tier=0 OpenHands/Trinity-Large-Thinking rec= 32.727273 delta= 0.027272727 tier=0 OpenHands/claude-fable-5 rec= 84.242424 delta= 0.042424242 tier=0 OpenHands/claude-opus-4-5 rec= 69.090909 delta= -0.009090909 tier=0 OpenHands/claude-opus-4-6 rec= 80.000000 delta= 0.0 tier=0 OpenHands/claude-opus-4-7 rec= 81.212121 delta= 0.012121212 tier=0 OpenHands/claude-opus-4-8 rec= 78.787879 delta= -0.012121212 tier=0 OpenHands/claude-sonnet-4-5 rec= 72.727273 delta= 0.027272727 tier=0 OpenHands/claude-sonnet-4-6 rec= 13.333333 delta= 0.033333333 tier=0 --- average_score (primary=AS_macro_raw_Dall, published range=[15.48, 81.0]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 43.998115 delta= -1.681884964 tier=div OpenHands/DeepSeek-V4-Pro rec= 39.232391 delta= -1.427608897 tier=div OpenHands/GLM-4.7 rec= 41.051235 delta= -1.208764647 tier=div OpenHands/GLM-5 rec= 50.499567 delta= 1.059566749 tier=div OpenHands/GLM-5.1 rec= 58.139031 delta= -0.100968742 tier=2 OpenHands/GPT-5.2 rec= 56.976331 delta= -1.863668675 tier=div OpenHands/GPT-5.2-Codex rec= 55.596541 delta= -2.683458728 tier=div OpenHands/GPT-5.4 rec= 64.295549 delta= 0.015549477 tier=1 OpenHands/GPT-5.5 rec= 65.923545 delta= -0.016455068 tier=1 OpenHands/Gemini-3-Flash rec= 47.568288 delta= -1.431711809 tier=div OpenHands/Gemini-3-Pro rec= 46.668980 delta= -2.371020102 tier=div OpenHands/Gemini-3.1-Pro rec= 57.807426 delta= -2.752574296 tier=div OpenHands/Gemini-3.5-Flash rec= 63.519505 delta= 0.879504587 tier=div OpenHands/Kimi-K2-Thinking rec= 38.849820 delta= -2.150179612 tier=div OpenHands/Kimi-K2.5 rec= 46.916525 delta= -2.263475205 tier=div OpenHands/Kimi-K2.6 rec= 57.152145 delta= 0.012144842 tier=1 OpenHands/MiniMax-M2.1 rec= 40.074449 delta= -1.085551 tier=div OpenHands/MiniMax-M2.5 rec= 43.554957 delta= -1.66504304 tier=div OpenHands/MiniMax-M2.7 rec= 41.693101 delta= -1.686898725 tier=div OpenHands/MiniMax-M3 rec= 57.288491 delta= 0.088491032 tier=2 OpenHands/Nemotron-3-Nano rec= 14.027255 delta= -1.452745098 tier=div OpenHands/Nemotron-3-Super rec= 35.424114 delta= -0.735886428 tier=div OpenHands/Qwen3-Coder-480B rec= 30.074399 delta= -0.865601014 tier=div OpenHands/Qwen3-Coder-Next rec= 41.314389 delta= -2.505611062 tier=div OpenHands/Qwen3.5-Flash rec= 37.817929 delta= -0.262070536 tier=2 OpenHands/Qwen3.6-Plus rec= 52.634708 delta= -0.225291606 tier=2 OpenHands/Trinity-Large-Thinking rec= 32.010535 delta= -0.049464623 tier=1 OpenHands/claude-fable-5 rec= 81.009504 delta= 0.009503732 tier=1 OpenHands/claude-opus-4-5 rec= 57.887124 delta= -2.692875736 tier=div OpenHands/claude-opus-4-6 rec= 64.046852 delta= -2.673148123 tier=div OpenHands/claude-opus-4-7 rec= 69.684588 delta= 0.024588351 tier=1 OpenHands/claude-opus-4-8 rec= 71.876698 delta= -0.003301841 tier=0 OpenHands/claude-sonnet-4-5 rec= 50.843959 delta= -2.156041123 tier=div OpenHands/claude-sonnet-4-6 rec= 42.668706 delta= -1.851293755 tier=div --- issue_resolution_cost (primary=C_mean_excl, published range=[0.0395, 2.1639]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 0.154627 delta= -0.005373389 tier=2 OpenHands/DeepSeek-V4-Pro rec= 0.039500 delta= -0.0 tier=0 OpenHands/GLM-4.7 rec= 0.512960 delta= -0.046639686 tier=2 OpenHands/GLM-5 rec= 1.042348 delta= -0.017652007 tier=2 OpenHands/GLM-5.1 rec= 1.538904 delta= -0.001095909 tier=1 OpenHands/GPT-5.2 rec= 0.833383 delta= -0.026616978 tier=2 OpenHands/GPT-5.2-Codex rec= 0.928647 delta= -0.01135267 tier=2 OpenHands/GPT-5.4 rec= 0.631618 delta= 0.001617873 tier=1 OpenHands/GPT-5.5 rec= 1.508292 delta= -0.011708496 tier=2 OpenHands/Gemini-3-Flash rec= 0.413713 delta= -0.006287146 tier=2 OpenHands/Gemini-3-Pro rec= 0.955117 delta= 0.005117257 tier=2 OpenHands/Gemini-3.1-Pro rec= 1.899574 delta= 0.029574152 tier=2 OpenHands/Gemini-3.5-Flash rec= 1.565151 delta= 0.035151037 tier=2 OpenHands/Kimi-K2-Thinking rec= 1.865360 delta= -0.134640434 tier=div OpenHands/Kimi-K2.5 rec= 0.405957 delta= -0.000342902 tier=1 OpenHands/Kimi-K2.6 rec= 0.662468 delta= -0.007532148 tier=2 OpenHands/MiniMax-M2.1 rec= 0.662398 delta= -0.049402468 tier=2 OpenHands/MiniMax-M2.5 rec= 0.306600 delta= 0.0 tier=0 OpenHands/MiniMax-M2.7 rec= 0.179500 delta= -0.0 tier=0 OpenHands/MiniMax-M3 rec= 0.153509 delta= -0.016491313 tier=2 OpenHands/Nemotron-3-Nano rec= 0.231805 delta= -0.008194897 tier=2 OpenHands/Nemotron-3-Super rec= 0.466300 delta= 0.0 tier=0 OpenHands/Qwen3-Coder-480B rec= 1.224379 delta= -0.035621342 tier=2 OpenHands/Qwen3-Coder-Next rec= 1.358974 delta= -0.00102587 tier=1 OpenHands/Qwen3.5-Flash rec= 2.163900 delta= 0.0 tier=0 OpenHands/Qwen3.6-Plus rec= 1.537900 delta= 0.0 tier=0 OpenHands/Trinity-Large-Thinking rec= 0.692633 delta= 0.002632792 tier=1 OpenHands/claude-fable-5 rec= 1.431896 delta= 0.001896229 tier=1 OpenHands/claude-opus-4-5 rec= 1.795308 delta= -0.024691707 tier=2 OpenHands/claude-opus-4-6 rec= 0.771950 delta= 0.00194993 tier=1 OpenHands/claude-opus-4-7 rec= 1.274306 delta= -0.055694053 tier=div OpenHands/claude-opus-4-8 rec= 0.741782 delta= -0.008218436 tier=2 OpenHands/claude-sonnet-4-5 rec= 1.150560 delta= -0.039440096 tier=2 OpenHands/claude-sonnet-4-6 rec= 1.027499 delta= -0.002501477 tier=1 --- frontend_cost (primary=C_mean_excl, published range=[0.19, 6.92]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 0.182011 delta= -0.007988844 tier=2 OpenHands/DeepSeek-V4-Pro rec= 0.374600 delta= 0.0 tier=0 OpenHands/GLM-4.7 rec= 0.655536 delta= -0.004463866 tier=1 OpenHands/GLM-5 rec= 0.793802 delta= 0.213802426 tier=div OpenHands/GLM-5.1 rec= 3.611553 delta= -3.308446717 tier=div OpenHands/GPT-5.2 rec= 2.512101 delta= -0.257898649 tier=div OpenHands/GPT-5.2-Codex rec= 2.563549 delta= -0.40645065 tier=div OpenHands/GPT-5.4 rec= 1.941333 delta= 0.491333206 tier=div OpenHands/GPT-5.5 rec= 3.952526 delta= 1.142526358 tier=div OpenHands/Gemini-3-Flash rec= 0.562134 delta= -0.237866035 tier=div OpenHands/Gemini-3-Pro rec= 1.499227 delta= 0.039226837 tier=2 OpenHands/Gemini-3.1-Pro rec= 1.327885 delta= 0.087884675 tier=div OpenHands/Gemini-3.5-Flash rec= 3.028048 delta= 1.038047578 tier=div OpenHands/Kimi-K2-Thinking rec= 2.124638 delta= -0.185361966 tier=div OpenHands/Kimi-K2.5 rec= 1.619280 delta= -0.002020496 tier=1 OpenHands/Kimi-K2.6 rec= 0.962958 delta= 0.322957907 tier=div OpenHands/MiniMax-M2.1 rec= 0.821321 delta= -0.582979047 tier=div OpenHands/MiniMax-M2.5 rec= 0.594894 delta= 0.000794457 tier=1 OpenHands/MiniMax-M2.7 rec= 0.388000 delta= 0.0 tier=0 OpenHands/MiniMax-M3 rec= 0.556376 delta= 0.206375514 tier=div OpenHands/Nemotron-3-Nano rec= 0.383568 delta= -0.016432256 tier=2 OpenHands/Nemotron-3-Super rec= 0.770501 delta= 0.002600719 tier=1 OpenHands/Qwen3-Coder-480B rec= 1.915779 delta= -0.174220771 tier=div OpenHands/Qwen3-Coder-Next rec= 1.603807 delta= 0.08380733 tier=div OpenHands/Qwen3.5-Flash rec= 2.621800 delta= -0.0 tier=0 OpenHands/Qwen3.6-Plus rec= 2.315800 delta= -0.0 tier=0 OpenHands/Trinity-Large-Thinking rec= 1.085672 delta= 0.345671748 tier=div OpenHands/claude-fable-5 rec= 6.584421 delta= 2.194421404 tier=div OpenHands/claude-opus-4-5 rec= 2.370218 delta= -0.169781621 tier=div OpenHands/claude-opus-4-6 rec= 2.414325 delta= 0.044324918 tier=2 OpenHands/claude-opus-4-7 rec= 4.477764 delta= 1.647764352 tier=div OpenHands/claude-opus-4-8 rec= 2.711281 delta= 0.901281239 tier=div OpenHands/claude-sonnet-4-5 rec= 1.828334 delta= -0.061665658 tier=div OpenHands/claude-sonnet-4-6 rec= 2.400069 delta= 0.160068991 tier=div --- greenfield_cost (primary=C_mean_excl, published range=[0.01, 12.49]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 0.652004 delta= 0.082003958 tier=div OpenHands/DeepSeek-V4-Pro rec= 0.157700 delta= 0.0 tier=0 OpenHands/GLM-4.7 rec= 1.879501 delta= 0.23950076 tier=div OpenHands/GLM-5 rec= 3.247666 delta= 1.017665937 tier=div OpenHands/GLM-5.1 rec= 6.531294 delta= 1.221293725 tier=div OpenHands/GPT-5.2 rec= 5.008580 delta= 0.30858008 tier=div OpenHands/GPT-5.2-Codex rec= 6.285028 delta= 0.78502805 tier=div OpenHands/GPT-5.4 rec= 4.038909 delta= -0.001091344 tier=1 OpenHands/GPT-5.5 rec= 5.562512 delta= 0.00251225 tier=1 OpenHands/Gemini-3-Flash rec= 2.989738 delta= 0.369738104 tier=div OpenHands/Gemini-3-Pro rec= 3.184242 delta= 0.00424155 tier=1 OpenHands/Gemini-3.1-Pro rec= 3.146106 delta= 1.9661061 tier=div OpenHands/Gemini-3.5-Flash rec= 9.738023 delta= 1.828023181 tier=div OpenHands/Kimi-K2-Thinking rec= 0.955313 delta= 0.415313411 tier=div OpenHands/Kimi-K2.5 rec= 1.348699 delta= 0.088699173 tier=div OpenHands/Kimi-K2.6 rec= 1.518957 delta= -0.001042614 tier=1 OpenHands/MiniMax-M2.1 rec= 0.379123 delta= -0.000876515 tier=1 OpenHands/MiniMax-M2.5 rec= 0.523828 delta= 0.033828224 tier=2 OpenHands/MiniMax-M2.7 rec= 0.892600 delta= 0.0 tier=0 OpenHands/MiniMax-M3 rec= 0.897841 delta= 0.277840655 tier=div OpenHands/Nemotron-3-Nano rec= 0.074085 delta= 0.044085017 tier=2 OpenHands/Nemotron-3-Super rec= 1.215300 delta= 0.0 tier=0 OpenHands/Qwen3-Coder-480B rec= 0.010000 delta= 0.0 tier=0 OpenHands/Qwen3-Coder-Next rec= 1.037370 delta= -0.002630303 tier=1 OpenHands/Qwen3.5-Flash rec= 4.330900 delta= 0.0 tier=0 OpenHands/Qwen3.6-Plus rec= 7.493600 delta= 0.0 tier=0 OpenHands/Trinity-Large-Thinking rec= 1.595176 delta= 0.395176126 tier=div OpenHands/claude-fable-5 rec= 12.491802 delta= 0.001801688 tier=1 OpenHands/claude-opus-4-5 rec= 11.216383 delta= 2.106382731 tier=div OpenHands/claude-opus-4-6 rec= 8.791388 delta= 1.101387911 tier=div OpenHands/claude-opus-4-7 rec= 6.502921 delta= 0.812920696 tier=div OpenHands/claude-opus-4-8 rec= 8.351037 delta= 0.521036683 tier=div OpenHands/claude-sonnet-4-5 rec= 3.450309 delta= 0.2203085 tier=div OpenHands/claude-sonnet-4-6 rec= 7.977369 delta= 1.497369208 tier=div --- testing_cost (primary=C_mean_excl, published range=[0.0354, 2.085]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 0.109114 delta= -0.010885855 tier=2 OpenHands/DeepSeek-V4-Pro rec= 0.035400 delta= 0.0 tier=0 OpenHands/GLM-4.7 rec= 0.370433 delta= 0.000433475 tier=1 OpenHands/GLM-5 rec= 0.908512 delta= -0.001488275 tier=1 OpenHands/GLM-5.1 rec= 0.757355 delta= -0.002644515 tier=1 OpenHands/GPT-5.2 rec= 0.541749 delta= -0.01825111 tier=2 OpenHands/GPT-5.2-Codex rec= 0.649546 delta= -0.010454244 tier=2 OpenHands/GPT-5.4 rec= 0.470347 delta= 0.000347415 tier=1 OpenHands/GPT-5.5 rec= 0.916439 delta= -0.003560891 tier=1 OpenHands/Gemini-3-Flash rec= 0.297946 delta= -0.002053555 tier=1 OpenHands/Gemini-3-Pro rec= 0.919761 delta= -0.090238543 tier=div OpenHands/Gemini-3.1-Pro rec= 1.635973 delta= 0.015973278 tier=2 OpenHands/Gemini-3.5-Flash rec= 1.025110 delta= -0.004889766 tier=1 OpenHands/Kimi-K2-Thinking rec= 1.241144 delta= -0.148855991 tier=div OpenHands/Kimi-K2.5 rec= 0.420554 delta= -0.004045646 tier=1 OpenHands/Kimi-K2.6 rec= 0.333774 delta= 0.003773964 tier=1 OpenHands/MiniMax-M2.1 rec= 0.522004 delta= -0.022095607 tier=2 OpenHands/MiniMax-M2.5 rec= 0.223900 delta= -0.0 tier=0 OpenHands/MiniMax-M2.7 rec= 0.128300 delta= 0.0 tier=0 OpenHands/MiniMax-M3 rec= 0.100387 delta= -0.009613156 tier=2 OpenHands/Nemotron-3-Nano rec= 0.180697 delta= -0.009302935 tier=2 OpenHands/Nemotron-3-Super rec= 0.333300 delta= -0.0 tier=0 OpenHands/Qwen3-Coder-480B rec= 0.919678 delta= -0.050322161 tier=div OpenHands/Qwen3-Coder-Next rec= 1.086524 delta= -0.013476207 tier=2 OpenHands/Qwen3.5-Flash rec= 1.710700 delta= 0.0 tier=0 OpenHands/Qwen3.6-Plus rec= 2.085000 delta= 0.0 tier=0 OpenHands/Trinity-Large-Thinking rec= 0.776700 delta= -0.023300298 tier=2 OpenHands/claude-fable-5 rec= 1.466528 delta= -0.003472357 tier=1 OpenHands/claude-opus-4-5 rec= 1.340141 delta= -0.039858575 tier=2 OpenHands/claude-opus-4-6 rec= 0.429927 delta= -7.2664e-05 tier=1 OpenHands/claude-opus-4-7 rec= 0.820535 delta= 0.000534791 tier=1 OpenHands/claude-opus-4-8 rec= 0.732638 delta= 0.002637991 tier=1 OpenHands/claude-sonnet-4-5 rec= 0.935999 delta= -0.044000565 tier=2 OpenHands/claude-sonnet-4-6 rec= 0.853401 delta= -0.016598715 tier=2 --- information_gathering_cost (primary=C_mean_excl, published range=[0.04, 7.91]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 0.065281 delta= 0.005280593 tier=2 OpenHands/DeepSeek-V4-Pro rec= 0.159400 delta= -0.0 tier=0 OpenHands/GLM-4.7 rec= 0.179758 delta= 0.029757751 tier=2 OpenHands/GLM-5 rec= 0.371426 delta= 0.01142645 tier=2 OpenHands/GLM-5.1 rec= 0.457668 delta= -0.012331558 tier=2 OpenHands/GPT-5.2 rec= 0.489365 delta= 0.00936452 tier=2 OpenHands/GPT-5.2-Codex rec= 0.630669 delta= 0.080668719 tier=div OpenHands/GPT-5.4 rec= 0.553417 delta= -0.056583025 tier=div OpenHands/GPT-5.5 rec= 0.681175 delta= -0.058824652 tier=div OpenHands/Gemini-3-Flash rec= 0.404695 delta= 0.024694744 tier=2 OpenHands/Gemini-3-Pro rec= 0.609886 delta= 0.109886297 tier=div OpenHands/Gemini-3.1-Pro rec= 2.080644 delta= 0.280643806 tier=div OpenHands/Gemini-3.5-Flash rec= 2.706491 delta= 0.076490663 tier=div OpenHands/Kimi-K2-Thinking rec= 0.680393 delta= 0.030392761 tier=2 OpenHands/Kimi-K2.5 rec= 0.378100 delta= 0.0 tier=0 OpenHands/Kimi-K2.6 rec= 0.410249 delta= -0.009750723 tier=2 OpenHands/MiniMax-M2.1 rec= 0.277393 delta= 0.030292605 tier=2 OpenHands/MiniMax-M2.5 rec= 0.058700 delta= -0.0 tier=0 OpenHands/MiniMax-M2.7 rec= 0.216300 delta= 0.0 tier=0 OpenHands/MiniMax-M3 rec= 0.317203 delta= -0.032797252 tier=2 OpenHands/Nemotron-3-Nano rec= 0.044400 delta= 0.004400017 tier=1 OpenHands/Nemotron-3-Super rec= 0.120600 delta= 0.0 tier=0 OpenHands/Qwen3-Coder-480B rec= 0.307220 delta= 0.027219816 tier=2 OpenHands/Qwen3-Coder-Next rec= 0.211322 delta= 0.081322385 tier=div OpenHands/Qwen3.5-Flash rec= 0.342200 delta= 0.0 tier=0 OpenHands/Qwen3.6-Plus rec= 0.337700 delta= -0.0 tier=0 OpenHands/Trinity-Large-Thinking rec= 0.284845 delta= -0.065155012 tier=div OpenHands/claude-fable-5 rec= 7.906199 delta= -0.003800982 tier=1 OpenHands/claude-opus-4-5 rec= 0.548708 delta= -0.001291706 tier=1 OpenHands/claude-opus-4-6 rec= 0.459490 delta= 0.019490411 tier=2 OpenHands/claude-opus-4-7 rec= 0.789040 delta= -0.100959892 tier=div OpenHands/claude-opus-4-8 rec= 1.160562 delta= -0.009438261 tier=2 OpenHands/claude-sonnet-4-5 rec= 0.874247 delta= 0.004247386 tier=1 OpenHands/claude-sonnet-4-6 rec= 0.406780 delta= -0.003219559 tier=1 --- average_cost (primary=AC_macro_raw, published range=[0.1533, 5.538]) --- OpenHands/DeepSeek-V3.2-Reasoner rec= 0.232607 delta= 0.012607293 tier=2 OpenHands/DeepSeek-V4-Pro rec= 0.153320 delta= 2e-05 tier=0 OpenHands/GLM-4.7 rec= 0.719638 delta= 0.043737687 tier=2 OpenHands/GLM-5 rec= 1.272751 delta= 0.244750906 tier=div OpenHands/GLM-5.1 rec= 2.579355 delta= -0.420644995 tier=div OpenHands/GPT-5.2 rec= 1.877036 delta= 0.003035573 tier=1 OpenHands/GPT-5.2-Codex rec= 2.211488 delta= 0.087487841 tier=div OpenHands/GPT-5.4 rec= 1.527125 delta= 0.087124825 tier=div OpenHands/GPT-5.5 rec= 2.524189 delta= 0.214188914 tier=div OpenHands/Gemini-3-Flash rec= 0.933645 delta= 0.029645222 tier=2 OpenHands/Gemini-3-Pro rec= 1.433647 delta= 0.01364668 tier=2 OpenHands/Gemini-3.1-Pro rec= 2.018036 delta= 0.476036402 tier=div OpenHands/Gemini-3.5-Flash rec= 3.612565 delta= 0.594564538 tier=div OpenHands/Kimi-K2-Thinking rec= 1.373370 delta= -0.004630444 tier=1 OpenHands/Kimi-K2.5 rec= 0.834518 delta= 0.016418026 tier=2 OpenHands/Kimi-K2.6 rec= 0.777681 delta= 0.061681277 tier=div OpenHands/MiniMax-M2.1 rec= 0.532448 delta= -0.125052206 tier=div OpenHands/MiniMax-M2.5 rec= 0.341585 delta= 0.006884536 tier=2 OpenHands/MiniMax-M2.7 rec= 0.360940 delta= 4e-05 tier=0 OpenHands/MiniMax-M3 rec= 0.405063 delta= 0.08506289 tier=div OpenHands/Nemotron-3-Nano rec= 0.182911 delta= 0.002910989 tier=1 OpenHands/Nemotron-3-Super rec= 0.581200 delta= 0.000500144 tier=1 OpenHands/Qwen3-Coder-480B rec= 0.875411 delta= -0.046588892 tier=2 OpenHands/Qwen3-Coder-Next rec= 1.059599 delta= 0.029599467 tier=2 OpenHands/Qwen3.5-Flash rec= 2.233900 delta= 0.0 tier=0 OpenHands/Qwen3.6-Plus rec= 2.754000 delta= 0.0 tier=0 OpenHands/Trinity-Large-Thinking rec= 0.887005 delta= 0.131005071 tier=div OpenHands/claude-fable-5 rec= 5.976169 delta= 0.438169196 tier=div OpenHands/claude-opus-4-5 rec= 3.454152 delta= 0.374151824 tier=div OpenHands/claude-opus-4-6 rec= 2.573416 delta= 0.233416101 tier=div OpenHands/claude-opus-4-7 rec= 2.772913 delta= 0.460913179 tier=div OpenHands/claude-opus-4-8 rec= 2.739460 delta= 0.281459843 tier=div OpenHands/claude-sonnet-4-5 rec= 1.647890 delta= 0.015889913 tier=2 OpenHands/claude-sonnet-4-6 rec= 2.533024 delta= 0.327023689 tier=div SECTION 4 — SELF-IDENTIFICATION (§3.1) AND EXECUTOR-ROW CELL CLASSES (symmetric) RULE: exact lower(language_model)=='claude-fable-5', fallback substring 'fable'; B2: startswith 'gemini' (pin unknown to this arm) A2 match type: exact == A2_rows == file_index0=0 file_rank=1 score_rank=1 {'id': 'OpenHands/claude-fable-5', 'language_model': 'claude-fable-5', 'sdk_version': 'v1.18.1', 'average_score': 81.0, 'average_cost': 5.538, 'categories_completed': 5, 'issue_resolution_score': 95.8, 'frontend_score': 70.6, 'greenfield_score': 62.5, 'testing_score': 91.9, 'information_gathering_score': 84.2, 'release_date': '2026-06-09'} == B2_candidate_rows == file_index0=18 file_rank=19 score_rank=19 {'id': 'OpenHands/Gemini-3-Pro', 'language_model': 'Gemini-3-Pro', 'sdk_version': 'v1.8.3', 'average_score': 49.04, 'average_cost': 1.42, 'categories_completed': 5, 'issue_resolution_score': 70.6, 'frontend_score': 36.8, 'greenfield_score': 25.0, 'testing_score': 68.6, 'information_gathering_score': 44.2, 'release_date': '2025-11-18'} file_index0=19 file_rank=20 score_rank=20 {'id': 'OpenHands/Gemini-3-Flash', 'language_model': 'Gemini-3-Flash', 'sdk_version': 'v1.8.3', 'average_score': 49.0, 'average_cost': 0.904, 'categories_completed': 5, 'issue_resolution_score': 74.6, 'frontend_score': 22.1, 'greenfield_score': 18.8, 'testing_score': 70.7, 'information_gathering_score': 58.8, 'release_date': '2025-12-22'} file_index0=6 file_rank=7 score_rank=7 {'id': 'OpenHands/Gemini-3.5-Flash', 'language_model': 'Gemini-3.5-Flash', 'sdk_version': 'v1.24.0', 'average_score': 62.64, 'average_cost': 3.018, 'categories_completed': 5, 'issue_resolution_score': 78.6, 'frontend_score': 39.7, 'greenfield_score': 37.5, 'testing_score': 76.2, 'information_gathering_score': 81.2, 'release_date': '2026-05-19'} file_index0=8 file_rank=9 score_rank=9 {'id': 'OpenHands/Gemini-3.1-Pro', 'language_model': 'Gemini-3.1-Pro', 'sdk_version': 'v1.11.5', 'average_score': 60.56, 'average_cost': 1.542, 'categories_completed': 5, 'issue_resolution_score': 76.8, 'frontend_score': 44.1, 'greenfield_score': 25.0, 'testing_score': 75.1, 'information_gathering_score': 81.8, 'release_date': '2026-01-15'} == cell classes for OpenHands/claude-fable-5 (addressed numeric columns, symmetric executor treatment) == issue_resolution_score class=reconciled_T0:S1_pct_Dall primary rec=95.8 delta=0.0 frontend_score class=reconciled_T0:S1_pct_Dall primary rec=70.58823529411765 delta=-0.011764706 greenfield_score class=reconciled_T0:S1_pct_Dall primary rec=62.5 delta=0.0 testing_score class=reconciled_T0:S1_pct_Dall primary rec=91.91685912240185 delta=0.016859122 information_gathering_score class=reconciled_T0:S1_pct_Dall primary rec=84.24242424242425 delta=0.042424242 average_score class=reconciled_T0:AS_macro_pub1dp primary rec=81.00950373178874 delta=0.009503732 issue_resolution_cost class=reconciled_T1:C_mean_excl primary rec=1.4318962289999988 delta=0.001896229 frontend_cost class=divergent primary rec=6.5844214044117635 delta=2.194421404 greenfield_cost class=reconciled_T1:C_mean_excl primary rec=12.491801687500002 delta=0.001801688 testing_cost class=reconciled_T1:C_mean_excl primary rec=1.4665276431870662 delta=-0.003472357 information_gathering_cost class=reconciled_T1:C_mean_excl primary rec=7.9061990181818205 delta=-0.003800982 average_cost class=divergent primary rec=5.976169196456131 delta=0.438169196 == cell classes for OpenHands/Gemini-3-Pro (addressed numeric columns, symmetric executor treatment) == issue_resolution_score class=reconciled_T0:S1_pct_Dall primary rec=70.6 delta=0.0 frontend_score class=divergent primary rec=24.752475247524753 delta=-12.047524752 greenfield_score class=reconciled_T0:S1_pct_Dall primary rec=25.0 delta=0.0 testing_score class=reconciled_T2:S1_pct_Dall primary rec=68.75 delta=0.15 information_gathering_score class=reconciled_T0:S1_pct_Dall primary rec=44.24242424242424 delta=0.042424242 average_score class=divergent primary rec=46.6689798979898 delta=-2.371020102 issue_resolution_cost class=reconciled_T1:C_mean_zero primary rec=0.9551172570281126 delta=0.005117257 frontend_cost class=reconciled_T2:C_mean_excl primary rec=1.4992268371134014 delta=0.039226837 greenfield_cost class=reconciled_T1:C_mean_excl primary rec=3.1842415499999994 delta=0.00424155 testing_cost class=divergent primary rec=0.9197614574074069 delta=-0.090238543 information_gathering_cost class=divergent primary rec=0.60988629689441 delta=0.109886297 average_cost class=reconciled_T2:AC_macro_raw primary rec=1.433646679688666 delta=0.01364668 == cell classes for OpenHands/Gemini-3-Flash (addressed numeric columns, symmetric executor treatment) == issue_resolution_score class=reconciled_T0:S1_pct_Dall primary rec=74.6 delta=0.0 frontend_score class=divergent primary rec=14.705882352941176 delta=-7.394117647 greenfield_score class=reconciled_T0:S1_pct_Dall primary rec=18.75 delta=-0.05 testing_score class=reconciled_T2:S1_pct_Dall primary rec=70.99767981438515 delta=0.297679814 information_gathering_score class=reconciled_T0:S1_pct_Dall primary rec=58.78787878787879 delta=-0.012121212 average_score class=reconciled_T2:AS_macro_raw_Dobs primary rec=47.56828819104102 delta=-1.431711809 issue_resolution_cost class=reconciled_T2:C_mean_excl primary rec=0.41371285399999963 delta=-0.006287146 frontend_cost class=divergent primary rec=0.5621339653846156 delta=-0.237866035 greenfield_cost class=reconciled_T1:C_mean_zero primary rec=2.9897381035714288 delta=0.369738104 testing_cost class=reconciled_T1:C_mean_excl primary rec=0.29794644477958243 delta=-0.002053555 information_gathering_cost class=reconciled_T1:C_mean_zero primary rec=0.4046947438311689 delta=0.024694744 average_cost class=reconciled_T2:AC_macro_raw primary rec=0.9336452223133591 delta=0.029645222 == cell classes for OpenHands/Gemini-3.5-Flash (addressed numeric columns, symmetric executor treatment) == issue_resolution_score class=reconciled_T0:S1_pct_Dall primary rec=78.6 delta=0.0 frontend_score class=divergent primary rec=40.298507462686565 delta=0.598507463 greenfield_score class=reconciled_T0:S1_pct_Dall primary rec=37.5 delta=0.0 testing_score class=reconciled_T2:S1_pct_Dall primary rec=76.38888888888889 delta=0.188888889 information_gathering_score class=divergent primary rec=84.81012658227849 delta=3.610126582 average_score class=divergent primary rec=63.51950458677079 delta=0.879504587 issue_resolution_cost class=reconciled_T2:C_mean_excl primary rec=1.5651510373493975 delta=0.035151037 frontend_cost class=divergent primary rec=3.02804757761194 delta=1.038047578 greenfield_cost class=reconciled_T1:C_mean_zero primary rec=9.73802318076923 delta=1.828023181 testing_cost class=reconciled_T1:C_mean_excl primary rec=1.025110233916084 delta=-0.004889766 information_gathering_cost class=reconciled_T2:C_mean_zero primary rec=2.7064906628289465 delta=0.076490663 average_cost class=divergent primary rec=3.61256453849512 delta=0.594564538 == cell classes for OpenHands/Gemini-3.1-Pro (addressed numeric columns, symmetric executor treatment) == issue_resolution_score class=reconciled_T2:S1_pct_Dall primary rec=77.0 delta=0.2 frontend_score class=divergent primary rec=30.392156862745097 delta=-13.707843137 greenfield_score class=reconciled_T0:S1_pct_Dall primary rec=25.0 delta=0.0 testing_score class=reconciled_T2:S1_pct_Dall primary rec=74.82678983833718 delta=-0.273210162 information_gathering_score class=reconciled_T0:S1_pct_Dall primary rec=81.81818181818181 delta=0.018181818 average_score class=divergent primary rec=57.807425703852815 delta=-2.752574296 issue_resolution_cost class=reconciled_T2:C_mean_excl primary rec=1.8995741519038059 delta=0.029574152 frontend_cost class=reconciled_T2:C_mean_zero primary rec=1.3278846752688174 delta=0.087884675 greenfield_cost class=reconciled_T1:C_mean_zero primary rec=3.1461060999999995 delta=1.9661061 testing_cost class=reconciled_T2:C_mean_excl primary rec=1.6359732775981524 delta=0.015973278 information_gathering_cost class=divergent primary rec=2.080643806060608 delta=0.280643806 average_cost class=divergent primary rec=2.0180364021662767 delta=0.476036402 SECTION 5 — OUTPUT ARTIFACT HASHES 6f6ebf6aa853bc5ebd21587c26f2770776adf210e4528224d2939cbb5000f7bc work/out_A_run1.json 6f6ebf6aa853bc5ebd21587c26f2770776adf210e4528224d2939cbb5000f7bc work/out_A_run2.json 4840a810430e9b3375c9e87ed81215e207f5c5ce76d3acf9aed1fe4c6b0dd243 work/out_B_run1.json 4840a810430e9b3375c9e87ed81215e207f5c5ce76d3acf9aed1fe4c6b0dd243 work/out_B_run2.json 3c50554de4c579e50cbf6a5be96c4c2a0e8d86ee7681ea1f06ac0008466bf21d work/impl_A.py d26b44da2ff67baf9ef9a0b4e6bfc87f6e151ad8a993a1bd41618ddc1ce72908 work/impl_B.py c7daf0a699e5c1e590823e8b0e75b87d4deb3e574948c17c9765eb0174a83ffc work/compare_impls.py END OF RESULTS