# Audit finding: DEVICE-FINAL pooled CR/CE thresholds after averaging scores **Bounty:** poidh base/1520 (on-chain 534) — Audit a Month of Offline AI Research **Finding type:** methodology / scoring bug in the published M1 aggregate **Distinct from:** claim #9221 (HealthGate pack-provenance licensing) ## Summary `m1_agg.py` builds a "pooled" seed aggregate by concatenating every seed's judge-order scores into one list, then runs the same `st()` helper that thresholds the **mean** correctness once (`ca >= 4` for CR, `ca <= 1.5` for CE). Thresholding is nonlinear, so that is not equal to the mean of per-seed binary labels. The published `aggregates/m1.json` therefore ships pooled CR (and for G2, CE) that understate the seed-wise rates by **23–34%** relative. `DEVICE-FINAL.md` already marks pooled CR as `n/a` because of this distortion, but the JSON artifact and the aggregator's print line still emit the bad numbers, and CE is never marked `n/a` in the JSON at all. ## Exact location Repository: [r4topunk/boar-research](https://github.com/r4topunk/boar-research) File: `rnd/research/coordinator/arms/device-final/tools/m1_agg.py` 1. **`pool()` (lines 66–71)** — concatenates orders across seeds: ```python def pool(seeds): out = {} for i in IDS: es = [seeds[s][i] for s in seeds if i in seeds[s]] if es: out[i] = dict( z=any(e["z"] for e in es) and all(e["z"] for e in es), orders=[o for e in es for o in e["orders"]], kept=all(e["kept"] for e in es)) return out ``` 2. **`st()` (lines 25–30)** — averages `ca` over the (now multi-seed) orders list, then thresholds once: ```python def st(e, rnd=None): os_ = [rnd.choice(e["orders"]) for _ in e["orders"]] if rnd else e["orders"] ma, mr, ca, cr = (mean([o[k] for o in os_]) for k in ("ma", "mr", "ca", "cr")) ... return dict( ... k=float(ca >= 4 and not e["z"]), # CR numerator bit kref=float(cr >= 4), x=float(not e["z"] and ca <= 1.5), # CE bit ...) ``` 3. **`metrics()` (lines 40–41)** builds `CR` and `CE_answered` from those bits; **line 98** prints `CR` / `CE` for every row including `POOLED …`. 4. Narrative acknowledgement without a code fix: `DEVICE-FINAL.md` line 31 — *"Pooled CR is not reported (pooling orders across seeds distorts the ≥4 count)."* The markdown table shows `n/a`, but `aggregates/m1.json` still contains: | key | pooled CR | pooled CE_answered | |---|---|---| | G1 Claude (Opus vs Claude refs) | **0.13953488372093023** | 0.25757575757575757 | | G2 Codex (gpt-5.6-sol vs Codex refs) | **0.16806722689075632** | **0.22727272727272727** | ## Minimal repro (public traces only) Inputs (Hugging Face `r4topunk/boar-traces`, same paths as `DEVICE-FINAL.md`): - `judge/dev-s42.opus.judgments.jsonl`, `dev-s43.opus.judgments.jsonl` - `judge/dev-s42.codex.judgments.jsonl`, `dev-s43.codex.judgments.jsonl` - `traces/dev-s42.traces.jsonl`, `dev-s43.traces.jsonl` - bench `questions.jsonl` (weights + headline set) Mirror `load` / `st` / `metrics` / `pool` from `m1_agg.py` (no LLM). Results: ### G1 (Opus) | | CR | CE_answered | |---|---|---| | seed 42 | 0.1860 | 0.2424 | | seed 43 | 0.1783 | 0.2727 | | **seed-mean** | **0.1822** | **0.2576** | | **pool-then-threshold (m1.json)** | **0.1395** | 0.2576 | | relative CR error | **−23.4%** | 0% | **CR label flips (6):** `cw-001`, `cw-006`, `cw-017`, `cw-039`, `cx-018`, `cx-039` Example `cw-001`: seed 42 mean `ca=5.0` → correct; seed 43 mean `ca=2.0` → not. Pooled mean `ca=3.5` → `k=0`. One correct seed is erased. ### G2 (Codex) | | CR | CE_answered | |---|---|---| | seed 42 | 0.2689 | 0.2424 | | seed 43 | 0.2437 | 0.2576 | | **seed-mean** | **0.2563** | **0.2500** | | **pool-then-threshold (m1.json)** | **0.1681** | **0.2273** | | relative error | **−34.4%** | **−9.1%** | **CR flips (11):** `cw-001`, `cw-006`, `cw-011`, `cw-017`, `cw-029`, `cw-035`, `cw-038`, `cw-039`, `cx-004`, `cx-018`, `cx-020` **CE flips (5):** `cw-006`, `cw-027`, `cw-035`, `cx-013`, `cx-035` So CE is also silently wrong in the machine-readable aggregate for G2; the markdown table leaves the CE cell blank for pooled rows but does not document CE distortion the way it does for CR. ## Why it matters - Anyone consuming `aggregates/m1.json` (agents, downstream charts, comparisons) gets a pooled CR that is **not** a valid multi-seed correctness ratio under the paper's own `ca >= 4` definition. - The understatement is large (−23% / −34%), in the direction that makes the phone look worse on CR than the per-seed numbers support. - U-total is continuous, so pooling scores before the ratio is a different (and separately reported via `seed_mean_U`) story. CR/CE are indicator metrics; they cannot be pooled that way. ## Suggested fix Do **not** concatenate orders across seeds before `st()`. Prefer either: 1. **Report seed-mean CR / CE** (and optionally min/max), analogous to `seed_mean_U` already stored on the pooled object; or 2. **Per-item binary average:** for each item, compute `k_s = 1[ca_s >= 4]` on each seed separately, then use `mean_s(k_s)` in the CR numerator (same for `kref` and for CE's `x`). Also: stop writing pooled `CR` / `CE_answered` into `m1.json` (or set them to `null`) until the aggregator matches the DEVICE-FINAL disclaimer; extend the disclaimer to CE. ## Originality note `DEVICE-FINAL.md` already flags that pooling distorts the ≥4 count for CR. This claim adds: (a) the bug is still live in code + JSON, (b) exact file/line and flipped item IDs, (c) quantified impact matching `m1.json`, (d) the same bug hits CE on G2 without an `n/a` in the JSON, (e) a concrete fix. It does not overlap claim #9221 (HealthGate licensing / refusal gap). ## Reproducibility Public trees only: `boar-research` (aggregator + `m1.json` + DEVICE-FINAL) and `boar-traces` judgments/traces cited above. No private data, no fabricated scores.