Did weighting or exclusion move the finding?
A worked example under the method — consumer / marketing research, held to the same three tests as every receipt.
Every consumer dataset passes through choices before it becomes a finding — weighting to a frame, imputing what's missing, dropping the inattentive. Each is defensible; any of them can move the result. The unanswered question is not whether the finding is true. It is whether a hand moved it.
That gap is not hypothetical. In “Many Analysts, One Data Set” (Silberzahn et al., 2018), 29 teams analysed the same data for the same question and returned odds ratios from 0.89 to 2.93 — a spread not explained by skill. Today the defense that “our own prep didn't make the effect” is procedural: the preregistration, the code, our word. There is no floor a skeptic can re-derive from the artifact. Here is that floor.
First, we tried to fool our own instrument
A caliper that rings on any big edit is worthless. So each real manipulation below is paired with an equal-magnitude placebo: drop the same number of rows at random, or apply the survey's own weights shuffled. What separates them is 21–32× — the structured prep step moves that many times further than its own placebo on all three datasets. The ring is a property of the structure that was moved, not of sample-size churn.
Two limits, stated up front rather than found later. First: that separation is the claim — not the word “holds.” A floor is a quantile of the data's own churn, and which quantile you declare is a choice. The placebos sit inside the 99th percentile on all three sets (0.3–0.7×), but against the median two of the three cross it. The separation, by contrast, is identical under either floor — 20.7× on Yogurt, 32.3× on GSS, 24.0× on NHANES — because the floor divides out of both sides. So we publish both floors and lead with the distance between the two operations, not with a pass/fail that moves when the quantile does. Second: this page used to tell you the geometry nominates the coupling you named — that it ranked your finding first among all cells, on its own. That claim is withdrawn, and the demonstration behind it was circular. The exclusion drops exactly the respondents who produce the finding, so the biggest move lands on that coupling by construction; ranking it afterwards and calling it a nomination was the instrument congratulating our own choice of operation. Run the identical mechanic on a decoy coupling nobody declared and the decoy crowns itself first. So: we name the target first, on domain grounds, and we claim only that it moved past its own floor. Whether a non-circular localization is possible — under an operation that does not target the coupling — we have not measured. It stays an open question rather than a selling point.
the exhibit — three tests, on the climate finding
corpus: GSS 2018 (US General Social Survey, n=2,179) · finding: political ideology × support for environmental spending (conservatives favour less, raw corr −0.305) · prep step tested: a defensible respondent exclusion. register: detection. Floor read two ways — the median and the 99th percentile of the coupling's own churn. Both are properties of this corpus and both converge as sampling effort grows, so both are declared and the numbers below are given against each.
The respondent file changed — its order changed. The geometry did not move, to the byte. A checksum flags the reshuffle; the geometry does not.
This red is a deliberate control — a declared exclusion run on purpose to prove the instrument rings on real structure; the random placebo of equal size does not.
Whether the geometry can nominate a coupling under an operation that does not target it is untested here, and therefore unclaimed. What this exhibit shows is hold and ring — and hold is the decidable one.
This card used to read “LOCALIZES — #1” and say the geometry crowns the coupling. Withdrawn. The exclusion above drops exactly the respondents who produce the finding, so the largest move lands on that coupling by construction — ranking it and calling the rank a nomination measured our own operation, not the field. The same shipped mechanic makes a decoy coupling crown itself too.
| exclusion | raw corr | │Δg│ | × median floor | × p99 floor |
|---|---|---|---|---|
| 0% | −0.305 | 0.0000 | 0.0× | 0.0× |
| 25% | −0.214 | 0.1011 | 5.6× | 1.5× |
| 50% | −0.096 | 0.2341 | 12.9× | 3.4× |
| 75% | +0.094 | 0.4707 | 25.9× | 6.9× |
| 90% | +0.279 | 0.7886 | 43.3× | 11.6× |
Floors, from 300 benign bootstrap resamples of this corpus: median 0.0182, p99 0.0682. Both are quantiles of the data's own churn and both converge as the resample count grows. At a 10% exclusion — below the curve shown — the two disagree (2.2× median, 0.6× p99): the smallest dose is exactly where declaring a different lawful quantile changes the answer, which is why both are printed rather than one. Why the floors read this way — the record.
recompute fingerprint — the geometry digit this dataset reduces
to, reproducible byte-for-byte from its own vectors on any engine that follows the kit’s published
canonicalization — not on any arithmetic that merely computes a covariance (see below) · sha256 =
f7d31d9e945971bd26076cf9b7c18901e9801cdaa373d45f2751f251a84ecc8b
The GREEN before/after above are
this same clean fingerprint; the RED is the identical recompute applied to the declared exclusion.
one receipt, three real datasets
The bar for a proof is real, pinned, public data — and more than one set. Here the same
non-influence receipt runs on three, each a finding a consumer researcher would report and a reviewer
would scrutinise. All three were cross-validated in R's canonical survey estimator to
three decimals.
| dataset | the finding | the prep step tested | rings (× median / × p99) | placebo (× p99) | verdict |
|---|---|---|---|---|---|
| Ecdat::Yogurt marketing · scanner panel |
Yoplait price × Yoplait choice across purchase occasions (corr −0.286 — a descriptive association, not a choice-model coefficient) | exclusion — erode the price effect toward zero | 53.2× / 14.0× | 0.68× — holds separation 20.7× | FIT |
| GSS 2018 climate attitudes |
conservatives favour less env. spending (corr −0.305) | exclusion — erode the ideology gap toward zero | 43.3× / 11.6× | 0.36× — holds separation 32.3× | FIT |
| NHANES weighted survey (CDC) |
the race–poverty association (unweighted corr +0.050, understated) | reweighting — the survey's OWN design weights | 22.2× / 7.1× | 0.30× — holds separation 24.0× | FIT |
Placebo (all three): an equal-size random drop, or shuffled weights, sits at 0.3–0.7× the p99 floor — and at 0.9–2.6× the median, i.e. two of the three cross it. The distance between the real step and its own placebo is what does not move: 21–32× under either floor. The ring survives which rows were dropped but dies when the magnitude is kept and the structure removed — so it is structure, not size. NHANES understates the association until its own weights are applied; applying them (a mandatory, legitimate step) rings 22.2× median / 7.1× p99 — the instrument sees that a real prep step moved the finding.
There used to be a “localizes” column here, reporting each finding's rank among all cells. It is gone. The exclusion selects rows by the coupling it then measures, so the coupling's rank was a restatement of our own choice of operation — on any of these datasets, and on decoy couplings we never declared. No rank on this page survives that, so no rank is published. The ratios above are unaffected: they compare a measured move to the data's own churn and never depended on the ranking.
the variables, as run
Every column, coding, and missing-data rule behind the three baseline numbers — enough to reconstruct each finding from the public dataset before touching any geometry.
| dataset | columns, as encoded | coding & subsetting | missing-data rule |
|---|---|---|---|
| Ecdat::Yogurt n = 2,412 purchase occasions |
chose_yoplait (choice=="yoplait"), price.yoplait,
price.dannon, feat.yoplait, chose_dannon
(choice=="dannon"); prices as shipped in the package |
the baseline is corr(chose_yoplait, price.yoplait) = −0.286
— point-biserial across occasions, descriptive; no choice model is estimated |
rows failing numeric parse dropped |
| GSS 2018 n = 2,179 complete cases |
polviews, natenvir, degree, age,
sex |
polviews 1 = extremely liberal … 7 = extremely
conservative; natenvir recoded +1 = spending too little on the
environment, 0 = about right, −1 = too much; baseline is
corr(polviews, natenvir) = −0.305 |
listwise deletion on polviews/natenvir/degree/age |
| NHANES::NHANESraw n = 20,293 → 16,428 complete cases |
indicators: black (Race1=="Black"), poor
(Poverty < 1.5), obese (BMI ≥ 30),
old (Age ≥ 50); weight WTMEC2YR — the
survey’s own |
baseline is unweighted corr(black, poor) = +0.050; the weighted
read applies WTMEC2YR, cross-checked against R’s
survey::svydesign with SDMVPSU/SDMVSTRA nesting, to
three decimals |
complete cases on Poverty/BMI/Age/WTMEC2YR.
A first pass silently coded NA poverty (arriving as NaN) as “not
poor”; R’s complete-case count caught it, the rule was fixed, and the numbers
then matched to the third decimal — the mismatch is part of this record |
the relief
Give a contested result a floor a reviewer recomputes rather than argues about: reorder the respondents and the geometry is byte-identical; run the actual weighting or exclusion and here is exactly how far the finding moved, graded against the data's own floor. Your design stands; your statistics stand. And the same object is a methods-literacy instrument for students: did your pipeline move your result? recompute it yourself makes the garden of forking paths physical in a way a lecture cannot.
what this does not prove
This page is a method demonstration on real public data. Held honestly:
- Non-influence is not validity. The receipt says whether the prep moved the finding — never that the finding is true, significant, or replicable. That reading stays yours.
- It does not find the coupling for you — and we withdrew the claim that it did. You name the coupling, on domain grounds, before anything runs. This page used to add that the geometry then independently nominated it — ranked it first among all cells, on its own. That was circular: the exclusion drops the rows that produce the coupling, so the biggest move lands there by construction, and a decoy coupling gets crowned the same way. The rank measured our operation. It is withdrawn, and no rank is published on this page. Whether a non-circular localization exists — under an operation that does not target the coupling — is untested and therefore unclaimed. The receipt claims that a named coupling moved past its own floor, and nothing more.
- The correlation collapse is partly definitional. “Drop the rows that produce the finding → the correlation falls” is near-tautological; the load is carried by the placebo, the data's own floor, and the geometry — not the raw correlation.
- The verdict depends on indicator encoding — the same encoding every shipped case uses, disclosed rather than presented as neutral. The in-tab check below is self-served; the independent version is the downloadable kit against a separately-published fingerprint.
The instrument is fixed. On three real consumer-research datasets it rang past each one's own floor when a real prep step moved the finding, and held when a same-size placebo did not — because the reading is conditioned by the corpus, and the floor is always the corpus's own. That is the method; this is the method at work.
verify this receipt in your browser
This page's receipt records its register, its verdict, and the headline geometry digest, and carries its own SHA-256. Recompute it here — WebCrypto, in your tab, nothing sent — and it matches; alter one character of a copy and it fails. It is the smallest version of the whole move: recompute, don't trust. The geometry digests themselves are recomputable on your bench from the kit.
published fingerprint ·
or download this receipt and check it on your bench:
receipt-consumer.json
— shasum -a 256 prints the same number. Individual case receipts are on
the ledger.
Want the full recompute kit — the vectors, a numpy-only verify.py, the
reproduce script, and the pinned public datasets? Request it by email
and it arrives by reply. Every number on this page falls out of the shipped geometry math on real,
public data, cross-validated in R.