# WS4 pre-registration: whole-dossier vs per-section vs per-claim step 03 validation

Written 2026-09-21 ~16:05 EDT, before any WS4 model call and before any WS4 model output exists.
Contract: EXPERIMENT-PLAN-jev-2026-09-21.md sections A-D and E/WS4 (frozen, not edited).
Known-defect labels (ground.py output) may be produced/read after this file is hashed; they are the answer key and
are never sent to a model. No held-out vendor ([vendor], [vendor]) is used for anything.

## Questions
- Q4.1 (primary) With evidence held constant (the vendor's full evidence.md, COHORT DIMS and rubric in every call),
  does mistral-small-3.2 catch a larger share of known defects when judging (c) one claim line per call than
  (a) the whole dossier in one call (production form)? Also (b) one `## ` section per call (secondary).
- Q4.2 Cost, tokens, latency per arm; cost per defect caught.
- Q4.3 Unneeded flags per arm on claim units with no known defect.

## Hypotheses (expected direction)
- H1 (primary): catch share (c) > (a). Rationale: one long call returns a short flag list and dilutes attention.
- H2: (b) lies between (a) and (c).
- H3: (c) raises more unneeded flags on clean units than (a) (a narrow excerpt lacks context, and the unchanged
  prompt asks for dim re-derivation the excerpt cannot support).
- H4: (c) costs the most per dossier; cost per defect caught is reported, no direction predicted.

## Sample
- Vendors: the 25 ys-glm-bench slugs minus [vendor] and [vendor] = 23 vendors (../ys-glm-bench/io/runs/).
- Drafts: dossier_incumbent.md (PRIMARY analysis); dossier_glm.md (SECONDARY, run only if budget allows after the
  primary is complete).
- Bundle per vendor: copied from ys-glm-bench/io/runs/<slug>/bundle/ into out/exp/ws4/io_<draft>/runs/<slug>/bundle/.
- Reduction rule (declared now): the per-claim arm is costed on 2 vendors first (the first 2 of the seeded order
  below). If the projected total for the PRIMARY (all 3 arms, incumbent drafts, 2 repeats) exceeds $4.00, vendors are
  dropped from the END of the seeded order until the projection is <= $4.00; all 3 arms always use the same vendor
  set. Seeded order = Python `random.Random(20260921).shuffle(sorted(slugs))`. The secondary glm run then uses the
  same seeded order and runs vendors while projected cumulative spend stays <= $4.75 (hard stop $5.00).
  The 2 pilot vendors' calls are real data (identical procedure), not discarded.

## Known defects (answer key; deterministic)
- Source: ground.py (worktree ../ys-jev-granularity-2026-09-21, origin/main cedc26f1), run offline by
  me with YS_ONBOARDING_IO inside out/exp/ws4/, each draft at runs/<slug>/stages/dossier.md. The existing
  results/per-vendor/*.json appear to hold counts rather than claim-level detail; if so ground.py is re-run.
  Incumbent and glm drafts use separate IO roots (io_incumbent, io_glm) so outputs never overwrite each other.
- Claim unit = one `[[`-opened provenance block as parsed by ground.parse_claim_units (identity: slug, draft, pos).
- Defect = a unit in ground's `ungrounded`, `mistagged` or `unresolved` lists. Clean = unit in `grounded_ok`.
  `auto_fixable` units (label flips) are in neither set and are excluded (count reported).
- Subsets reported separately: mistagged-only defects; non-mistag defects (ungrounded/unresolved), and ungrounded by
  `why` reason.
- Secondary label source: claims_200 dev-split claims for [vendor] and [vendor], located in the incumbent draft by the
  rule below; their internal_verdict (Opus-dominated) is compared with each arm's catch.

## Arms (prompt = production user_03 from run_stages.py on origin/main, verbatim; SYS_03 verbatim)
Only the DOSSIER UNDER REVIEW text changes:
- (a) whole dossier: the full draft text.
- (b) per section: the draft's first `# ` title line (if any) + blank line + one `## ` section (heading line through
  the line before the next `## ` heading). Only sections containing >= 1 claim unit are sent. Text before the first
  `## ` that contains units is sent as its own section.
- (c) per claim line: the title line + blank line + the section heading line + blank line + the single full source
  line that contains the unit. One call per distinct line holding >= 1 unit (a table row with several `[[` blocks is
  one call covering all units on it).
- Prompt sha256: the sha256 of the user_03 TEMPLATE rendered with a fixed placeholder dossier ("<<DOSSIER>>") per
  vendor, identical across arms by construction, is recorded per arm; per-call full-prompt sha256 is logged per row.
- Parameters as production gh_chat: model mistralai/mistral-small-3.2-24b-instruct, temperature 0.0,
  response_format json_object, max_tokens 8000 (production retries an empty completion with a doubled budget, cap
  32000; replicated), no provider pinning; provider served is logged. 2 repeats per arm per unit of work.
  Workers <= 3. API errors retried with backoff, at most 4 tries, logged, never counted as an answer.
- Parse: production extract logic (strip fence, parse JSON); a parse failure is recorded as verdict "error" and is
  NOT an answer (counted separately).

## Catch rules (declared before any model call)
Flagged strings F(call) = every entry of `ungrounded_or_mistagged_claims` plus every entry of `flags` (non-string
entries are JSON-serialised). Normalisation N(x): lowercase, unwrap web.archive.org/web/<ts>/ prefixes, drop
`http(s)://` and `www.`, drop trailing `/`, collapse whitespace.
- Unit claim text L(u) = the full source line containing the unit.
- Content tokens T(x) = lowercase alphanumeric tokens of length >= 4 not in a fixed stopword list (in match.py),
  plus all numbers.
- A flagged string s MATCHES unit u if any of:
  (M1) URL: some N(url of u), with length >= 12, is a substring of N(s);
  (M2) receipt: u has a receipt and the first min(30, len) chars of N(receipt) appear in N(s), or a quoted
       fragment of s (text between quotes, >= 15 chars after N) appears in N(receipt) or N(L(u));
  (M3) wording: |T(s) & T(L(u))| >= 4 and |T(s) & T(L(u))| / |T(s)| >= 0.5.
  Score(s,u) = 1000*M1 + 1000*M2 + |T(s) & T(L(u))|. Among the units IN THAT CALL's text that s matches, s credits only
  the unit(s) with the maximum score (ties all credited; tie-credits counted and reported).
- Arm (a) and (b) catch of unit u in one repeat: some s in F(call covering u) credits u.
- Arm (c) catch of unit u in one repeat (PRIMARY rule for (c)): the call covering u's line returns verdict != "pass"
  (errors excluded), consistent with plan section B "Mistral catch". If the line holds >1 unit, all units on the
  line share that call's verdict (multi-unit lines counted and reported).
- Sensitivity (exploratory): (c) under the flagged-string rule (same as (a)/(b)); (a)/(b) under the call-verdict
  rule (a unit counts as caught if the call covering it is non-pass; for (a) this is the dossier verdict).
- Conservative 2-repeat rule: a unit is caught in an arm if it is caught in ANY repeat that returned a parsed answer.
  If both repeats error for a unit's call, the unit is excluded from comparisons involving that arm (reported).
- Dossier-level verdict per arm: (a) the call verdict; (b)/(c) "fail" if any call in the dossier is non-pass.
  Reported per repeat and under the any-repeat rule.

## Metrics
- PRIMARY (Q4.1): share of known defects (incumbent drafts, all defect types) caught, (c) minus (a), paired by defect.
  Wilson 95% CI per proportion; vendor-clustered bootstrap 95% CI for the difference (2,000 resamples, seed 20260921,
  resampling vendors with replacement); exact two-sided McNemar on discordant defects.
- Secondary: (b) vs (a), (c) vs (b) (same statistics, exploratory); mistag-only and non-mistag subsets; glm drafts;
  dossier-level verdicts; Q4.2 cost/prompt tokens/completion tokens/latency per arm (total, per dossier, per defect
  caught); Q4.3 unneeded-flag share on clean units per arm with Wilson CI and the paired (c)-(a) difference with
  cluster bootstrap and McNemar; error/parse-failure rates; provider distribution.
- Opus-label secondary: claims_200 claims with split == "dev" and vendor in {[vendor], [vendor]} are located at the unit u
  in the incumbent draft maximising |T(claim) & T(L(u))| (plus 3 if claim evidence_url matches a unit URL under M1);
  located iff that overlap is >= 4 tokens and >= 0.4 of |T(claim)|. For located claims: each arm's catch vs
  internal_verdict (recorded fail caught / recorded pass flagged). Descriptive only (n expected < 10 per cell).
- Cells with n < 10 are flagged "too small to read".

## Exclusions
- [vendor], [vendor] (held-out). Any vendor whose bundle is missing evidence.md/meta.json/rubric.md or where ground.py
  fails: excluded and logged. Units with no parsed answer in both repeats of an arm: excluded from that arm's pairs.
- Nothing else is excluded. No threshold, rule or prompt is tuned after seeing outputs; any change is a logged
  deviation.

## Seeds
- Vendor order and bootstrap: 20260921. Model calls are temperature 0 (no seed parameter sent, as production).
