# WS1 pre-registration: Jev shadow pilot on fresh dossiers

Written 2026-09-21 ~20:05Z, before any model call and before joining any labels or model outputs. Contract:
`EXPERIMENT-PLAN-jev-2026-09-21.md` (sha256 ca31341b...ab2d), sections A-D and E/WS1. Any later change is logged in
`LOG.md` as a deviation with its reason.

## Questions and hypotheses

Q1.1 On fresh step 02 drafts, how many claims per dossier are in scope for Jev, and how often do `ground.py` and step 03
flag them?
- Primary metric: in-scope claim units per dossier (per-dossier count, plus mean and range across dossiers).
- Secondary: share of in-scope claims flagged by ground.py (claim unit appears in `defects.json` claim-level defect
  lists: ungrounded / mistagged / receipt-not-on-page / not-in-bundle), with Wilson 95% CI; share of in-scope claims
  whose text or URL appears (raw string match) in step 03 `ungrounded_or_mistagged_claims` or `flags`; step 03
  dossier verdict per run; skip reasons for out-of-scope units.
- Expectation (no directional test): 20-80 claim units per dossier; most sourced units in scope; ground.py flag
  share under 30%.

Q1.2 On the same live claims, does Jev accept more when the evidence is the verbatim window located in the saved page
(arm B) than when it is the dossier's own receipt text (arm A)?
- H1 (directional): accept share B > accept share A. Reason: the arm-B window contains the receipt plus surrounding
  context, and it is taken from the page itself, so it holds the text the claim paraphrases more often than the short
  receipt does.
- Primary metric: paired accept-share difference (B minus A) on in-scope claims, where "accept" is the frozen current
  rule (policy A, threshold 0.8: flag `none` and confidence >= 0.8) under the conservative 2-repeat rule (accepted only
  if both repeats accept). Test: exact two-sided McNemar on the discordant pairs. Interval: vendor-clustered bootstrap
  95% CI of the difference (2,000 resamples of vendors with replacement, seed 20260921).
- Secondary (exploratory): per-arm accept share with Wilson CI; per-repeat (single-call) accept share; derived-fail
  share (checker view) per arm; flag distribution per arm; the difference split by arm-B evidence kind (verbatim
  located window vs `keyword_window` fallback) and by tag (VENDOR-CLAIMED vs THIRD-PARTY); repeat agreement per arm.

Q1.3 What does a re-draft cost, what does the sidecar cost, and how long does each take?
- Primary metric: USD per re-draft (driver run) from the pipeline's own cost reporting (stage outputs / run
  summary) if present, otherwise from OpenRouter `usage.cost`; reported per stage (01, 02, 03) and total.
- Secondary: wall-clock seconds per driver run and per stage where recorded; sidecar USD and seconds per dossier;
  sidecar USD per in-scope claim.

Q1.4 Projection: how many dossiers and how much blind adjudication to estimate Jev's miss rate to within +-5 and +-3
percentage points (95%) at the flag rates seen in the pilot?
- Primary metric: number of dossiers needed for +-5 pp and +-3 pp, under the normal-approximation half-width
  z*sqrt(p(1-p)/n_fail) <= h with z = 1.96, for miss-rate p in {0.05, 0.10, 0.20, 0.50 (worst case)}.
  n_fail recorded fails needed -> adjudicated claims = n_fail / f, where f = pilot fail-base-rate proxy -> dossiers =
  claims / (in-scope claims per dossier). Primary f proxy: share of in-scope claims flagged by ground.py OR step 03
  (union). Sensitivity: f from ground.py alone, from step 03 alone, from Jev arm-B checker view; design effect
  1.0 / 1.5 / 2.0 for vendor clustering (and a pilot ICC-based design effect if estimable, exploratory).
- Adjudication load reported two ways: (i) adjudicate every in-scope claim; (ii) adjudicate only the claims needed
  to reach n_fail (same as (i) when fails are not pre-identifiable).

## Sample

- Frozen-bundle vendors (fixed): [vendor], [vendor], [vendor], [vendor], [vendor]. Bundle copied from
  `../ys-eval-bundle/io/runs/<slug>/bundle/`, driver run with `--no-refresh`.
- Fresh-retrieval vendors: up to 5, in ascending order of sha256("shadow-1" + slug) over slugs in
  `content/vendor-research/*.md` on origin/main (worktree at cedc26f1), excluding every claims-dataset vendor, every
  `ys-eval-bundle` slug and every `ys-glm-bench` slug. Stop fresh retrieval after 2 failed runs. If fresh retrieval
  needs something unavailable (e.g. a Chrome DevTools endpoint), do not install or launch anything system-level;
  record it and proceed with the 5 frozen-bundle vendors only.
- For every slug, record tune/confirm half = int(sha256("shadow-1" + slug), 16) mod 2 (0 = tune, 1 = confirm).
- One driver run per vendor (no re-runs to get a pass). Runs one at a time.

## Units, scope and arms

- Claim unit: one `[[`-opened provenance block from `ground.parse_claim_units` (imported from the worktree). Claim id =
  section + index within the dossier. Claim text = the text on the same line before the `[[` block, back to the
  previous `]]` or table-cell boundary, markdown stripped (fallback: ground.py `context`).
- In scope: at least one sourced tag (VENDOR-CLAIMED or THIRD-PARTY) AND at least one cited URL whose page is in
  `bundle/sources_index.json` with `ok` true (matched with ground.py's own URL normalization). All other units are
  logged as `skipped` with the reason (no sourced tag / no URL / URL not in bundle / page not ok).
- Arm A evidence: the receipt text the dossier gives for that claim (`receipt: "..."`). If the unit has no receipt,
  arm A evidence is the literal string "(The dossier gives no receipt text for this claim.)" and the claim stays in
  the primary analysis (it reflects what step 02 wrote); a secondary analysis restricts to units with a receipt.
- Arm B evidence: the receipt located in the saved page text (normalized whitespace/case/quote match) plus 2
  sentences either side, `evidence_kind=verbatim_window`; if not found (or no receipt), the `t2_raw_excerpt.py` large
  keyword window from the saved page, `evidence_kind=keyword_window`.
- Jev call: the `claims_probe.ask` request shape (7-flag vocabulary, RULES) unchanged, model `~typesafe/jev-latest`,
  2 repeats per arm, full probabilities kept, as_of = the dossier run date. Section = the dossier `##` heading. Rubric
  text: none (live claims carry no rubric row; noted as a caveat). Current rule frozen (threshold 0.8).

## Exclusions

- API or parse errors: retried with backoff (max 4 tries), logged, never counted as an answer. A claim with an
  errored repeat in either arm after retries is excluded from the primary paired analysis and reported separately.
- Driver runs that never reach step 03 are excluded from the sidecar (logged with exit code). Runs that reach step 03
  are included pass or fail (exit 5 is normal).
- No other exclusions. Duplicate claim units (same text and URL) are kept in the primary analysis; a de-duplicated
  count is reported as a sensitivity check.

## Statistics

- Proportions: Wilson 95% CI. Differences/ratios: vendor-clustered bootstrap 95% CI, 2,000 resamples, seed 20260921.
  Paired binary: exact McNemar (two-sided binomial on discordant pairs). n on every number. Cells n < 10 flagged
  too small to read. One primary metric per question; everything else exploratory.
- Held-out: no claims-dataset vendor is used (none are sampled), so no held-out data is touched.

## Adjudication packet

- `adjudication_packet.jsonl`: every in-scope claim unit: random packet id, claim text, section, vendor, source URL,
  arm-B evidence, saved-page path, as_of. No model outputs, no ground.py or step 03 results. Shuffled with
  random.Random(20260921); packet ids = 8 hex chars from the same generator. Key in `adjudication_key.json`.

## Spend and stop rules

- Cap $10 total for WS1 (driver + sidecar). Stop if the first re-draft costs more than $1.50.
- Stop on: any write outside the worktree / `out/exp/ws1/`; any driver step that touches D1, R2, email or git; 3
  consecutive failed runs (a run that fails to reach step 03).
