# WS2 pre-registration: Jev trade-off curve and confidence calibration

Written 2026-09-21 before any label was joined to a Jev output. Contract: `EXPERIMENT-PLAN-jev-2026-09-21.md`
(sha256 ca31341b...ab2d), sections A-D and E/WS2. No API calls; spend cap $0.

What was inspected before writing this (shapes only, no join): row keys of `results_{dev,test}.json`; 4 rows of
`id, rep, flag, confidence, probs`; `probs` holds only options with p >= 0.005, rounded to 3 decimals, so a missing
option means p < 0.005 and is read as 0; `confidence` is a separate number, 2 decimals, a little below the top
probability; flag equals argmax(probs) on 514/516 dev and 227/228 test calls; 0 API errors; every claim has exactly 2
repeats. Dataset: dev 258 claims / 85 vendors (171 recorded pass, 87 recorded fail, per plan section A and the file's
own stats block); test 114 claims / 41 vendors; no vendor in both splits. Label producers from `label_provenance.json`
(counts only, not joined).

## Data, unit, exclusions

- Inputs: `out/claims_200/results_dev.json`, `results_test.json`, `claims_200.json`, `label_provenance.json`.
- Answer key: `internal_verdict` from `claims_200.json` (authoritative). The `key_verdict` copy inside the results
  rows is checked against it; any mismatch is reported and the claims file wins.
- Exclusions: calls with `error` set (none seen); claims without both repeats (none seen) are dropped from 2-repeat
  scores only; claims whose recorded verdict is not `pass`/`fail` (none expected) are dropped. Counts reported.
- Unit: the claim for every rule, score and curve (2 repeats combined as stated per score). The call (claim x repeat)
  for calibration, with a claim-level sensitivity (repeat 0 only).
- Fail-flag set (plan section B): `dim_disagreement_major`, `contradicted_fact`, `fabricated_or_unsupported_claim`.
  Minor flags derive verdict pass but are not an accept under the current rule.

## Seeds and resampling

- Folds: 5-fold vendor-grouped CV on dev. fold(vendor) = int(sha256(("ws2-2026-09-21" + vendor).encode()).hexdigest(), 16) mod 5.
  Fold sizes (claims, vendors) are reported. The same folds serve every candidate.
- Bootstrap: vendor-clustered, 2,000 resamples, numpy default_rng(20260921); resample vendors with replacement within
  the split, carry all their claims (and both repeats). Percentile 95% intervals.
- Proportions: Wilson 95% (z = 1.96). Paired binary comparisons: exact two-sided McNemar (binomial on discordant pairs).
  Cells under n = 10 are flagged as too small to read.

## Q2.1 Is Jev's confidence calibrated?

- Primary metric: expected calibration error (ECE) of the `confidence` field for calls whose flag is `none`, outcome =
  claim is a recorded pass, dev, both repeats pooled. 10 equal-width bins on [0, 1] (bin i = [i/10, (i+1)/10), last bin
  closed). ECE = sum_b (n_b / N) * |pass share_b - mean confidence_b|. 95% CI by vendor-clustered bootstrap.
- Hypothesis H2.1 (direction): Jev is overconfident on `none`: signed gap (mean confidence minus recorded-pass share)
  > 0, and ECE >= 0.05. Reading rule, fixed now: ECE < 0.05 well calibrated; 0.05-0.10 moderately miscalibrated;
  > 0.10 poorly calibrated. "Overconfident" is claimed only if the bootstrap CI of the signed gap excludes 0.
- Secondary: (a) same ECE for fail-flag answers, outcome = recorded fail (H: overconfident, gap > 0);
  (b) reliability tables per bin with n, mean confidence, observed share, Wilson CI; (c) signed gap with CI;
  (d) repeat-0-only ECE.
- Exploratory: P(none) in place of `confidence`; minor-flag answers (share recorded pass by bin); ECE split by
  `evidence_is_verbatim` and by label producer (Opus validator vs other).
- Held-out (test): the same ECE, reliability table and signed gap, plus the existing rule's operating point (below),
  reported descriptively and labelled HELD-OUT, DESCRIPTIVE. Nothing is chosen from it.

## Q2.2 Does a probability-based score beat the current rule at zero dev misses?

Scores (higher = more accept-worthy; accept iff score >= threshold). Per repeat r:
`cn_r` = confidence if flag is `none` else 0; `pn_r` = P(none) (0 if absent); `pnf_r` = 1 - sum of the 3 fail-flag
probabilities (absent = 0).

| id | score | repeats |
|---|---|---|
| C0 | min_r cn_r (current rule family; C0 >= 0.8 is exactly the current conservative 2-repeat rule) | 2 |
| K1 | cn_0 (current rule, 1 repeat) | 1 |
| K2 | min_r pn_r | 2 |
| K3 | mean_r pn_r | 2 |
| K4 | pn_0 | 1 |
| K5 | min_r pnf_r | 2 |
| K6 | mean_r pnf_r | 2 |
| K7 | pnf_0 | 1 |

Threshold picking on a training set, fixed now: candidate cut points are the midpoints between adjacent unique
training score values, plus one below the minimum (accept all) and +infinity (accept none). The zero-miss threshold is
the most permissive cut with 0 training recorded fails accepted. Miss-capped thresholds (secondary) are the most
permissive cut with training miss rate (misses / training recorded fails) <= cap.

- Primary metric: pooled out-of-fold accept share of dev recorded passes (n = 171) for (i) the candidate procedure and
  (ii) C0, each with thresholds picked at zero training misses on the 4 other folds. The candidate procedure is nested:
  in each outer fold it picks, from K1-K7, the score with the highest training accept share of recorded passes at zero
  training misses (ties broken by the table order), then applies that score and its threshold to the held-out fold.
  Reported: both accept shares with Wilson CIs, the difference (candidate minus C0) with a vendor-clustered bootstrap
  CI over the fixed out-of-fold decisions, exact McNemar on recorded passes, and out-of-fold misses (of n = 87) for
  both, with Wilson CIs and McNemar on recorded fails.
- Hypothesis H2.2 (direction): candidate >= C0, a small gain; expected CI to include 0. A gain is claimed only if the
  bootstrap CI of the difference excludes 0 AND the candidate's out-of-fold misses are not higher than C0's.
- Pre-stated degenerate case: if a recorded fail in a training set carries a top score, the zero-miss cut accepts
  nothing. If both arms accept 0 out of fold, the primary answer is "no Jev score reaches zero dev misses with any
  accepts"; that is reported as the result, not tuned around.
- Secondary: (a) the same procedure at training miss caps of 5% and 10%; (b) the primary excluding provenance-only
  recorded fails (`prov_only`, a mis-tag / provenance label Jev cannot see) from the miss count; (c) accept share of
  ALL dev claims (not only recorded passes); (d) the fixed current rule (C0 >= 0.8, no tuning) on dev.
- Exploratory: each score K1-K7 alone under the same CV (all reported, so no hidden selection); which score each outer
  fold selected.
- "Best candidate" for 2.3 = the score the nested procedure selected in most outer folds (tie: higher pooled
  out-of-fold accept share at zero training misses, then table order).

## Q2.3 Trade-off curves and adjudication sample sizes

- Curves (dev): for C0 and the best candidate, sweep every cut point over the full dev set; at each, miss rate
  (misses / 87) and accept share of recorded passes (/ 171), each with Wilson CI. This is a description of dev, not a
  choice; the out-of-fold operating points from 2.2 are overlaid.
- Operating points: for C0 and the best candidate, three CV operating points (training miss cap 0%, 5%, 10%), plus the
  current rule as deployed (C0 at 0.8). Assumed miss rate p at each = the pooled out-of-fold dev miss rate; a
  conservative variant uses p = its Wilson upper bound.
- Precision target: 95% interval half-width d = 0.03 and 0.05. Adjudicated fails needed:
  normal approximation n = ceil(1.96^2 p (1 - p) / d^2) (degenerate at p = 0: reported but flagged); Wilson: smallest n
  with max(upper - p, p - lower) <= d at observed share p.
- Adjudication load: claims to adjudicate blind = ceil(n / base rate), at the dev recorded-fail base rate (87/258) and
  at 0.10 and 0.20 as stated assumptions (the live base rate is WS1's to measure).
- Hypothesis H2.3 (direction): the curve is steep near zero misses (moving from the 10% cap to zero misses costs most
  of the accept share).

## Analysis order

1. Build score tables (no labels). 2. Join labels. 3. 2.1 dev, then test descriptive. 4. 2.2 CV. 5. 2.3 curves and
sample sizes. 6. charts.json and REPORT.md. Deviations are logged in LOG.md with the reason.
