# WS3 pre-registration: Jev error map and complementarity with step 03

Written 2026-09-21 ~16:05 ET, before any claim type was tagged and before any label, evidence field, label-provenance
record or model output was opened. Before writing this I looked only at: plan sections A-D and E/WS3; field NAMES of
`claims_200.json`, `results_*.json`, `step03_*.json`, `label_provenance.json`; the distinct values of the non-label
claim attributes `section`, `cohort`, `evidence_kind`, `split`, `evidence_is_verbatim`; and the `FLAGS`/`RULES`/verdict
code in `claims_probe.py`. No API calls; spend cap $0.

## 1. Questions and primary metrics (one per question)

- Q3.1 Where does Jev miss or over-flag? PRIMARY: Jev miss rate in the checker view, by claim type, on recorded fails,
  pooled dev + held-out (dev and held-out also shown separately, held-out descriptive only).
- Q3.2 Are Jev's and Mistral's misses correlated? PRIMARY: odds ratio of the 2x2 table Jev miss (checker view) x
  Mistral miss (vocab variant) on recorded fails, pooled dev + held-out, with a vendor-clustered bootstrap 95% interval,
  compared with 1.0 (what independence predicts).
- Q3.3 Why was each recorded fail missed by either model? PRIMARY: distribution of cause codes (section 5) over the
  set of recorded fails missed by Jev (checker view) or Mistral (vocab), pooled, each a count with a Wilson interval.

## 2. Hypotheses (expected direction, stated before looking)

- H1 (3.1): Jev's checker-view miss rate is higher for `score_rubric`, `numeric_fact` and `customers_logos` claims
  (precision and judgment defects) than for `integration` and `certification_compliance` claims (the item is listed in
  the evidence or it is not). Expected: score_rubric miss rate above the pooled miss rate.
- H2 (3.1): Jev misses more often on paraphrased evidence (`evidence_is_verbatim` false) than on verbatim evidence,
  because a paraphrase can smooth over the defect. Expected: paraphrase miss rate > verbatim miss rate.
- H3 (3.1): Fails labelled by the deterministic detector (`ground.py`) are caught more often than fails labelled by the
  Opus validator, because detector defects are string-level mismatches visible in the evidence. Low confidence.
- H4 (3.2): Misses are positively correlated (odds ratio > 1): both models read the same claim and evidence, so a
  defect that is not visible in the evidence escapes both. Expected: OR > 1, P(Mistral misses | Jev misses) >
  P(Mistral misses | Jev catches).
- H5 (3.2): Unneeded escalations on recorded passes are also positively correlated (OR > 1).
- H6 (3.3): At least 30% of misses are coded `NOTVIS` (defect not visible in the supplied evidence) or `QLABEL`
  (questionable label), i.e. not a clear model error.

## 3. Definitions (from plan section B, made operational)

- Recorded fail/pass: `internal_verdict` in `claims_200.json`.
- Jev per repeat: accept = flag `none` AND confidence >= 0.8 (policy A, threshold 0.8), recomputed from `flag` and
  `confidence`; the stored `escalate` field is NOT used (it may reflect another threshold). Checker verdict = fail if
  flag in {`dim_disagreement_major`, `contradicted_fact`, `fabricated_or_unsupported_claim`}, else pass.
- Conservative 2-repeat rules (both models):
  - Jev escalation view: a claim is accepted only if every valid repeat accepts. Miss (escape) = recorded fail
    accepted. Unneeded escalation = recorded pass not accepted.
  - Jev checker view: the claim is judged fail if ANY valid repeat's checker verdict is fail. Miss = recorded fail
    judged pass in every valid repeat. Over-flag (unneeded escalation, checker view) = recorded pass judged fail in any
    valid repeat.
  - Mistral (step 03 per claim): catch = `verdict != "pass"` in ANY valid repeat. Miss = recorded fail with every valid
    repeat `pass`. Unneeded escalation = recorded pass with any valid repeat non-pass. The stored `caught` field is not
    used; it is recomputed.
- Valid repeat: a row with no `error` and (Jev) a flag in FLAGS / (Mistral) a non-empty verdict. Errored repeats are
  dropped and counted; a claim is judged on its remaining valid repeat(s). A claim with no valid repeat for a model is
  excluded from that model's metrics and from joint tables, and reported. Sensitivity (exploratory): Jev escalation
  view with errors treated as escalations (the production fail-closed rule).
- Vocab variant is primary for Mistral; prod is secondary.

## 4. Claim-type taxonomy (tagged from claim text alone, before any outcome is opened)

One primary type per claim. When more than one applies, the FIRST in this precedence order wins. The type is decided
by the claim's central assertion (what would have to be true for the claim to be true), not by incidental words.

1. `score_rubric`: the claim states a Yardstick score, rating, rank, tier, rubric level, dimension value, penalty,
   fit verdict or other evaluative judgment produced by the rubric (for example "scores 4 on AI-native", "scale fit:
   mid-market", "weak integration depth").
2. `pricing`: price, fee, plan or package, billing unit, free trial or freemium, discount, contract term, pricing
   transparency.
3. `certification_compliance`: certifications, attestations and audits (SOC 2, ISO 27001, HITRUST, PCI DSS, FedRAMP),
   regulatory compliance (HIPAA, GDPR, CCPA, FINRA), and named security controls (encryption, SSO/SAML, MFA, data
   residency, penetration testing, BAA).
4. `integration`: a connection to a named third-party product or platform (integrates with X, connector, marketplace
   or app-store listing, sync, API link to X), or a statement about the integration catalogue.
5. `customers_logos`: named customers, customer or user counts, case-study outcomes attributed to a customer,
   testimonials, customer segments served by name.
6. `funding_company`: company facts: founding, headquarters, founders and executives, headcount, funding rounds,
   investors, valuation, revenue, acquisitions and ownership, parent company, legal status, office locations.
7. `date_event`: a dated product event not covered above (launch, release, announcement, rebrand, end of life), where
   the date or sequence is the central assertion.
8. `numeric_fact`: a quantitative performance, outcome, scale or usage statistic not covered above (percent
   improvement, volume processed, uptime, time saved, number of languages or modules when the count is the point).
9. `capability_feature`: what the product does or supports: features, modules, AI functions, channels, platforms,
   deployment options, workflows.
10. `other`: anything else (positioning, target market, product category, awards, descriptive statements).

Secondary deterministic tag (not a type): `has_number` = the claim text contains a digit or a spelled number/amount
token (regex over digits, "%", "$", "million", "billion"). Reported as an exploratory dimension.

Method record per claim (`method` field): `rule` (a deterministic keyword rule fired and I kept its answer after
reading the claim), `manual_override` (a rule fired and I changed it on reading), `manual_no_rule` (no rule fired; my
reading). Rules are keyword regexes applied in the precedence order above; the script is saved as `tag_rules.py`.

## 5. Miss-cause code list (Q3.3), one code per missed recorded fail

- `PROV`: provenance-tag-only defect (the fact is supported; the fail is about a missing or wrong evidence tag,
  source attribution or tier label).
- `NUM`: numeric over-specification or numeric mismatch (a number more precise, larger or different than the evidence).
- `QUAL`: non-numeric qualifier or scope overreach ("all", "native", "fully", "leading", an extra product or market).
- `STALE`: stale fact (true earlier, superseded; the evidence or label note says so).
- `NOTVIS`: defect not visible in the supplied evidence (the supplied evidence plausibly supports the claim; the
  recorded fail rests on outside knowledge, a re-fetch, or material not in the `evidence` field). No model is blamed.
- `QLABEL`: questionable label (the supplied evidence clearly supports the claim and the label's own basis looks
  wrong, inconsistent or mis-transcribed).
- `RUBRIC`: rubric or score judgment (the disagreement is about a score, rating or fit level).
- `VISIBLE`: the defect is plainly visible in the supplied evidence (contradiction or clear absence) and the model
  missed it; a clear model error not covered by a more specific code above.
- `OTHER`: none of the above (one-line reason given).

Precedence when several apply: `QLABEL` > `NOTVIS` > `PROV` > `STALE` > `RUBRIC` > `NUM` > `QUAL` > `VISIBLE` >
`OTHER` (label and evidence problems first, because they decide whether a model can be blamed at all). Coding uses the
claim, the `evidence` field, the label's recorded basis/note and flag, and the models' outputs. One line per claim.

## 6. Sample and exclusions

- All 372 claims (dev 258, held-out 114) for tagging; recorded fails (125) for miss analyses; recorded passes (247) for
  over-flag / unneeded-escalation analyses. Exclusions only as in section 3 (no valid repeat). No claim is dropped for
  any other reason. Held-out results are described and pooled where the plan says, never used to choose anything;
  nothing in this workstream is chosen (no thresholds, rules or cut points are tuned).

## 7. Analysis

- Q3.1: per claim type, section, cohort, evidence form (`evidence_is_verbatim`), label producer
  (`label_provenance.json` producer) and `has_number`: Jev miss rate (checker view; primary for claim type) and Jev
  escape rate (escalation view), each with n and Wilson 95% interval, over recorded fails; over-flag rate (checker view)
  and unneeded-escalation rate (escalation view) over recorded passes. Pooled, then dev and held-out separately. Cells
  with n < 10 are flagged "too small to read". Exploratory difference paraphrase minus verbatim (checker miss rate)
  with a vendor-clustered bootstrap 95% interval.
- Q3.2: 2x2 tables on recorded fails (Jev checker miss x Mistral vocab miss; primary pooled). Report
  P(M miss | J miss), P(M miss | J catch) with Wilson intervals, the expected both-miss count under independence
  (row total x column total / n), the odds ratio (Haldane-Anscombe +0.5 on every cell if any cell is 0, applied the same
  way in every bootstrap replicate) with a vendor-clustered bootstrap 95% percentile interval, and an exact McNemar
  test comparing the two models' miss rates on the same fails. Secondary: prod variant; Jev escalation view; dev and
  held-out separately; the same analysis for unneeded escalations on recorded passes (Jev escalation view primary
  there for "unneeded escalation", checker over-flag secondary). Fisher exact p (ignores clustering) is exploratory.
- Bootstrap: resample vendors (`vendor` field) with replacement, 2,000 resamples, numpy seed 20260921; percentile
  interval; replicates with an undefined statistic are dropped and their count reported.
- Q3.3: one line per recorded fail missed by either model (Jev checker view or Mistral vocab): id, split, which
  model(s) missed, code, short reason. Counts per code with Wilson intervals, split by who missed (Jev only, Mistral
  only, both).
- Blind re-tag: at the end, re-tag 30 claims drawn with `random.Random(3303).sample(sorted ids, 30)` from claim text
  only, without opening `claim_types.json`; report percent agreement and Cohen's kappa. Caveat recorded: by then I will
  have seen labels and outputs, so the re-tag is blind to the first tags, not to outcomes.

## 8. Seeds and files

Bootstrap seed 20260921; re-tag sample seed 3303. Inputs: `out/claims_200/{claims_200,results_dev,results_test,
step03_vocab_dev,step03_vocab_test,step03_prod_dev,step03_prod_test,label_provenance}.json` (sha256 logged in LOG.md).
