# F4 pre-registration: how often are Jev's auto-accepts bad? (build half)

Written 2026-09-22 about 02:40Z, before any model call in F4 and before any F4 claim or Jev output exists.
Approval: founder, 2026-09-22, plan section F.4, spend cap $5 (OpenRouter). Method reused from WS1
(`../ws1/PREREG.md`, `../ws1/REPORT.md`, `../ws1/scripts/`, `../ws1/sidecar/`); differences are listed at the end.
Any later change is logged in `LOG.md` as a deviation with its reason. This build half never sees review labels; the
orchestrator runs the blind review on `adjudication_packet.jsonl` afterwards.

## Question and endpoints

Primary endpoint. Of the in-scope live claims Jev auto-accepts under the fixed production rule, what share are bad?
- Rule (fixed by founder decision, not tuned): arm B evidence (a window of text from the saved cited page, WS1's
  arm B), Jev `~typesafe/jev-latest` through the WS1 sidecar request shape, 2 repeats; accept = flag `none` AND
  confidence >= 0.8 on BOTH repeats. An errored repeat (after the harness's 4 tries) is never an accept.
- "Bad" = the blind reviewer's fail verdict on the packet line (defined by the orchestrator's review protocol, not
  here).
- Estimate: bad arm-B accepts / all arm-B accepts, with a Wilson 95% interval. Target n >= 100 arm-B accepts, so that
  0 bad would bound the share at about 3% (Wilson upper bound 3.7% at 0/100, 3.4% at 0/110).

Secondary (all descriptive, no tests):
1. Arm A (the draft's own receipt text as evidence): bad share of arm-A accepts, Wilson 95%.
2. Fail share in the random sample of the non-accepted in-scope claims (claims accepted by neither arm), Wilson 95%.
3. Estimated share of all bad in-scope claims that arm B accepts:
   bad_B_accepts / (bad claims among the fully reviewed accepts of either arm + N_rest * p_rest), where N_rest is the
   number of non-accepted in-scope claims and p_rest is the fail share in their random sample (weight N_rest / n_s).
   Interval: vendor-clustered bootstrap, 2,000 resamples of drafts with replacement, seed 20260922, weights recomputed
   per resample. Same quantity reported for arm A.
4. Catches by the existing checks on reviewed fails: of the reviewed fails (accepts of either arm plus the random
   sample), how many ground.py flagged (ungrounded / mistagged, as in WS1) and how many step 03 matched (raw string
   match of claim text, receipt or URL in `ungrounded_or_mistagged_claims` or `flags`, as in WS1). Random-sample
   fails are also reported weighted.
5. Counts: drafts, in-scope claims per draft, arm-A and arm-B accepts, packet size, spend.

## Units and scope (unchanged from WS1)

- Claim unit: one `[[`-opened provenance block from `ground.parse_claim_units` (imported read-only from the scratch
  worktree). Claim text: the text before the block on its line (table rows: the non-empty cells before the tag cell,
  joined with " | "; fallback ground.py context when under 12 characters).
- In scope: a sourced tag (VENDOR-CLAIMED or THIRD-PARTY) and at least one cited URL whose page is in
  `bundle/sources_index.json` with ok true. Everything else is logged as skipped with its reason.
- Arm A evidence: the dossier's receipt text, or the fixed note "(The dossier gives no receipt text for this claim.)".
- Arm B evidence: the receipt located verbatim in the saved page plus 2 sentences either side (cap 4000 characters),
  else the `t2_raw_excerpt` large keyword window from the best-matching ok cited page.
- Jev: `claims_probe.ask` shape unchanged (7-flag vocabulary, RULES), 2 repeats per arm, 3 worker threads, no rubric,
  as_of = the UTC date of the run.

## Sample, vendor order and exclusions

- Source: the 38 frozen bundles in `../ys-eval-bundle/io/runs/<slug>/bundle/`, copied (not moved) into
  `f4/onboarding-io-p<pass>/runs/<slug>/bundle/` and drafted with the production driver
  (`scripts/agent/onboarding/onboard_vendor.py --slug <slug> --no-refresh`) in the scratch worktree
  `../ys-jev-shadow-2026-09-21` (detached at cedc26f1), models as the pipeline itself chooses them.
- Excluded: `[vendor]` and `[vendor]` (held-out vendors) and `[vendor]` (it has claims in the test split of
  `claims_200.json`; only the vendor and split fields were read). 35 vendors are eligible.
- Order: ascending sha256("f4-2026-09-22" + slug), with WS1's five ([vendor], [vendor], [vendor], [vendor], [vendor]) last in
  their own hash order. The order and halves are in `sample.json` (`scripts/select_f4.py`):
  [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor],
  [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor],
  [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], [vendor], then [vendor], [vendor], [vendor], [vendor], [vendor].
- Tune / confirm half per vendor: int(sha256("f4-2026-09-22" + slug), 16) mod 2 (0 tune, 1 confirm). Recorded in
  `sample.json` and in the key for future threshold work only. Nothing in F4 is tuned.
- One driver run per vendor per pass, one at a time, no re-runs to get a pass. Pass 2 (a second draft of each vendor,
  same order, fresh copy of the frozen bundle in `onboarding-io-p2`) runs only if pass 1 ends short of the target.
- Included runs: every run that produces `stages/dossier.md` and `stages/defects.json` (driver exits 0, 1 or another
  final-gate code, 4 and 5). Runs that reached step 03 are tagged `primary`; exit 4 runs (ground.py H-types, no step
  03) are tagged `secondary`. Both are pooled for the primary endpoint and the stop count (Jev's accept decision is
  made per claim); the tag goes into the key so a step-03-only sensitivity can be computed. A run with no dossier
  (crash, API failure, lint fail, stub exit 2, cohort-fit halt exit 3) is a failed run and gets no sidecar.
- Duplicate claims (same text, same URL, e.g. across passes) are kept; a de-duplicated count is reported.

## Blind packet

- Selected: every in-scope claim accepted by arm A or arm B at 0.8 (conservative 2-repeat rule), plus a random sample
  of the other in-scope claims of size ceil(0.10 * N_rest), drawn with random.Random("f4-sample-2026-09-22") from the
  rest sorted by (run_id, unit_index).
- Shuffled together with random.Random("f4-packet-2026-09-22"); packet ids are 8 hex characters from that generator.
- Packet fields (as WS1): packet_id, vendor, section, claim, source_url, evidence (arm B window), saved_page_path,
  as_of. Nothing in the packet shows Jev's output, ground.py or step 03 results, or why a claim was selected.
- Key (`adjudication_key.json`, never shown to the reviewer): packet id -> run_id, pass, slug, half, claim_id,
  unit_index, claim_sha256, arm_b_sha256, set (primary / secondary), and selected_because (accept_A, accept_B,
  accept_AB, random_sample, with N_rest and n_s for the weight).

## Stop rules

- Stop after the vendor whose sidecar brings the pooled arm-B accepts to 110 or more.
- Spend: measured as the change in `usage` from `GET https://openrouter.ai/api/v1/key` since the start of F4
  (baseline 67.228502 at 2026-09-22T02:34:02Z). Another experiment shares the key, so this overcounts (the safe
  direction). Before each vendor run: if spend so far plus a $0.25 reserve would pass $5.00, stop.
- Stop after two passes.
- Safety stops: the worktree is not clean after the restore; any write outside `f4/` and the worktree's own dossier
  path; 3 consecutive failed runs.
- The worktree is restored after EVERY run as WS1 did (`git checkout -- content/vendor-research/<slug>.md`, since the
  driver's gate-fail path moves the tracked dossier into `onboarding-io/rejected/`), then `git status -s` is recorded.

## Differences from WS1

- New salt, order and vendors; WS1's five are last. Pooled primary set (WS1 kept exit 4 out of its McNemar primary).
- The packet holds accepts plus a 10% random sample, not every in-scope claim, and the key records why each line was
  selected.
- Spend measured on the key's usage delta, not per-stage price snapshots (those are still logged from usage.json).
