# Reproducibility package: Jev as a router for dossier claim review

This package accompanies the research article that evaluates "Jev" (TypeSafe's
claim classifier, called through OpenRouter) as a router: a model that decides
which claims in a vendor-research dossier can skip a second review. It holds
the pre-registrations, the analysis and scoring scripts, anonymised claim-level
results and the aggregate results files for every workstream the article reports
on. It does not hold the dossiers, the claims or the evidence text.

Everything here was produced from the original experiment folders by the build
scripts described in `RELEASE-REPORT.md`. Nothing was hand-edited after the
build. Vendor identities are replaced by stable codes (`V001` to `V153`); the
order of the codes was shuffled with a fixed seed and is not alphabetical.

## Layout

| folder | what it holds |
|---|---|
| `prereg/` | redacted copies of each workstream's pre-registration or protocol, plus `HASHES.md` |
| `scripts/` | the analysis, scoring and chart scripts, plus `t2_raw_excerpt.py` (the evidence-window code) and `recount.py` |
| `data/` | claim-level results, one CSV per set |
| `results/` | aggregate results files and the rendered charts, vendor names removed |

Run `python scripts/recount.py` from this folder to recompute the headline
counts from `data/` alone.

## Data files

| file | rows | one row per | source set |
|---|---|---|---|
| `data/pilot.csv` | 125 (110 in scope) | claim unit in the live pilot | ws1 |
| `data/scaleup.csv` | 576 (480 in scope) | claim unit in the live scale-up | f4 |
| `data/devset_calls.csv` | 744 | Jev call on the historical development set (two repeats per claim) | ws3, re-keyed answer key with 120 recorded fails |
| `data/devset_claims.csv` | 372 | claim on the historical development set, with both checkers' verdicts | ws3 |
| `data/ws4_planted.csv` | 2237 | claim unit in the planted-defect study (incumbent, planted and glm drafts) | ws4 |
| `data/ws4_plants.csv` | 69 | planted defect (type, ground category, confirmed), text dropped | ws4 |
| `data/founder_f3.csv` | 40 (26 kept) | founder-labelled pilot claim: founder verdict next to the blind Opus verdict | adj/founder |

Columns shared by the two live files: `set`, `vendor_code`, `claim_seq`
(sequential id), `claim_id` (dossier section and position), `section`,
`scope` and `skip_reason`, `tags` (provenance tag on the claim),
`ground_status` and `ground_flag` (the grounding check), `step03_*` (the
whole-dossier validator), `arm_a_kind` and `arm_b_kind` (evidence kind sent
to Jev on each arm: `keyword_window`, `verbatim_window`, `none_found`,
`receipt`, `no_receipt`), `keyword_score` (keyword overlap of the window;
recorded for every keyword window in both live runs), `jev_{A,B}_r{1,2}_flag` and `_conf` (Jev's
flag and confidence per arm and repeat), `accept_{A,B}_conservative` and
`fail_view_{A,B}_any` (the accept rule and the checker view), then the Opus
adjudication (`opus_verdict`, `opus_flag`, `opus_confidence`,
`opus_supported_by_cited_source`, `opus_true_per_web`), the reliability
re-label (`opus_reliability_*`), the whole-page re-read
(`reread_window_*`) and the spliced-window re-read (`reread_spliced_*`).
Empty cells mean the claim was not in that adjudication packet.

A claim counts as "bad" in the article when `opus_verdict` is `fail` or
`opus_reliability_verdict` is `fail`. That definition reproduces every
pilot and scale-up figure the article reports (9 bad of 110, 3 bad of 37
sampled, and the checker-view counts).

In the development-set files, `key_verdict` and `key_flag` are the re-keyed
answer key (120 recorded fails). In `devset_calls.csv` the five claims ruled
not fails on 2026-09-22 carry `key_verdict` `pass` and `key_flag`
`rekeyed_pass_2026-09-22`, so the two files agree; `jev_*` are Jev's outputs; `mv_*` and `mp_*` are the comparison
checker's verdicts under its two prompt variants (see
`scripts/devset/ws3_analyze.py`). `key_flag` values that were free text have
been replaced by `free_text_note_redacted` because the notes named vendors.

## How the files map to the article's findings

1. **Finding 1, Jev's accepts were clean as a router.** `data/pilot.csv` and
   `data/scaleup.csv` (`accept_B_conservative` against `opus_verdict`),
   `results/pilot_adjudication.json`, `results/scaleup_adjudication.json`,
   `data/founder_f3.csv` and `results/founder_f3_SCORE.md` for the founder's
   human check, charts `adj-*`, `f4-*`, `ws1-*`.
2. **Finding 2, it clears about a quarter of checkable claims, only fact rows.**
   `data/devset_calls.csv`, `data/devset_claims.csv`,
   `results/devset_ws2_results.json` (calibration and the accept curves),
   `results/devset_tables.json`, charts `ws2-*`.
3. **Finding 3, as a second checker it adds little.** `data/devset_claims.csv`
   (`jev_checker_fail` next to `mv_flagged` and `mp_flagged`),
   `results/devset_cause_codes.json`, `results/devset_explor_or_robustness.json`,
   charts `ws3-*`.
4. **Finding 4, the weak point was upstream of Jev.** `data/ws4_planted.csv`,
   `data/ws4_plants.csv`, `results/ws4_results.json`, the `ground_*` and
   `step03_*` columns of the two live files, `results/pilot_analysis.json`,
   charts `ws4-*`.
5. **Finding 5, faithful to the cited source is not the same as true.**
   `opus_supported_by_cited_source` against `opus_true_per_web` in the live
   files, the `reread_*` columns, and
   `results/evidence_channel_measure_armb.out` (what the arm-B evidence window
   actually contained).

Finding 6 (evidence text can steer a validator) is on hold and is not in this
package; see below.

## What was left out, and why

- **Claim text, evidence text, URLs, saved-page paths and vendor identities.**
  The claims describe the vendors and would identify them. Claim hashes are
  also dropped, because a hash of the text lets someone confirm a guess.
  Reviewer notes, quotes, free-text flag notes and run receipts are dropped for
  the same reason. The code-to-vendor key exists only in the build folder
  `_private/`, which is marked DO NOT PUBLISH and is not part of this package.
- **The prompt-injection tests (ws5 and ws10).** These are on hold pending the
  vendor's right of reply and are excluded entirely: no data, scripts, charts
  or pre-registrations from them are here.
- **The Opus review transcripts.** The adjudication was done by Claude Opus
  reviewers reading the claim and the evidence; their reasons and quotes repeat
  that text. Only the verdict, flag, confidence and the two support fields are
  released.
- **Narrative reports (`REPORT.md` files) and the evidence-channel write-up.**
  They quote claims and page text next to vendor names. The numbers they
  report are in `results/`.
- **Vendor-selection scripts** (`select_fresh.py`, `select_f4.py`) and the
  claim-typing scripts of ws3. The first two exist only to pick vendors from a
  private bundle and carry the vendor list in code; the ws3 typing scripts
  work on claim text, which is not released. Shell wrappers that only set up
  local folders are also left out.
- **The experiment plan** (`EXPERIMENT-PLAN-jev-2026-09-21.md`). It names
  vendors throughout; its hash is recorded in `prereg/HASHES.md`.

## Pre-registrations are redacted

The copies in `prereg/` have vendor names replaced with `[vendor]`, absolute
paths made relative and usernames removed. Hashing a copy will therefore not
reproduce the value published for it. `prereg/HASHES.md` lists, for each
original, the sha256 recorded in the workstream log at launch (before any
model call) and whether the original still matches that hash today.

## Scripts

Every script in `scripts/` is the original with absolute paths made relative,
usernames removed and any hard-coded vendor list replaced by a load from an
environment variable or data file (`WS4_SECONDARY_VENDORS`,
`WS4_PLANT_SEED_VENDOR`, `HELD_OUT_VENDORS`, `EXTRA_VENDOR_SLUGS`,
`HELD_OUT_CAPTION_REGEX`, `T2_SLUGS_FILE`). The scripts that call the model
read `OPENROUTER_API_KEY` from the environment and never print it; no key
material is in this package. Most scripts read the original experiment
folders, so they document the method rather than run on `data/` directly;
`scripts/recount.py` is the one written for this package and runs on `data/`.