Testing Jev in High Volume Retrieval and Routing

The short version

Yardstick Research builds evidence-tagged dossiers on software vendors. A model pipeline drafts them, and each claim in a draft cites a source page. Whether a claim can be trusted comes down to retrieval: was the right passage pulled from the cited page, and does that passage say what the claim says? This research measured that retrieval step and asked whether a small, cheap classifier, given the retrieved passage, can decide which claims are safe to accept and which ones need a heavier review. The classifier under test is Jev, a typed-choice model from TypeSafe, served through OpenRouter.

• Testing Jev as a router on live claims. Across two blind-reviewed live runs, Jev auto-accepted 127 claims. The reviewer judged none of them bad. A claim is bad when the blind reviewer, a Claude Opus agent that saw the claim, the saved copy of its cited page and the scoring rubric but never any model's output, gave it one of the three failing findings (a major score disagreement, a fact the page contradicts, or a claim the page does not support) or recorded that the claim is not true according to any public source; every other claim is good. Zero out of 127 still leaves an upper bound: the 95% interval runs from 0 to about 2.9%. The reviewer was Claude Opus, not a person, and one Jev version was tested at one point in time.
• Testing how many claims Jev can route past review. It accepts about a quarter of the claims it can check, and only fact rows. In the larger run it accepted 111 of 480 checkable claims (23%). It accepted no score rows, because it is not given the scoring rubric.
• Testing Jev as a second checker beside the existing validator. As a second checker, it adds little. When it is used to call claims pass or fail, the recorded fails it does not flag are largely the same ones the LLM validator already in the pipeline (Mistral Small 3.2 24B, as configured in Yardstick's pipeline) does not flag. The two are not independent checks.
• Testing Jev against known false claims. About 8 to 10% of the claims Jev did not accept were judged bad by the blind reviewer. The pipeline's existing claim checks flagged none of them. Most drafts did not quote their sources, so nothing compared a claim with the words on the cited page. The samples are small: 9 bad claims in the first live run (the pilot, 110 claims) and 3 in the sample from the second live run (the scale-up, 480 claims), so those zeros carry 95% upper bounds of 30% and 56%.
• Testing whether a claim the page supports is a true claim. Faithful to the cited source is not the same as true. Some claims matched their cited page and were wrong in the world today. Others were true, but the page they cited did not show it. A source-support check cannot detect the first kind.
• Testing Jev against prompt injection. Prompt injection here means a sentence planted in the evidence that tells the reviewer the claim has been checked and to pass it. Under that text, Jev's accept rule accepted none of 156 planted fail cases (95% upper bound 2.4%). The text did change Jev's answers, toward review: it accepted 9 of 41 recorded passes on evidence with no planted text and 0 to 7 of 41 with planted text. That shows Jev did not accept a planted fail case under these strings, not that Jev cannot be made to accept one.

Each finding below states what was tested, what happened and what it means, with its caveats. The next steps are collected under What happens next.

Background

The pipeline

Yardstick's onboarding pipeline drafts a vendor dossier from a frozen bundle of saved web pages, in five steps. Each claim in a draft carries a provenance tag (for example VENDOR-CLAIMED or THIRD-PARTY) and the URL of the page it came from. The validator reads the whole dossier, the evidence digest and the scoring rubric in one call.

The classifier

Jev is a typed-choice classifier. It does not write text and cannot browse. It reads a small JSON object (here, one claim and a piece of evidence) and picks one answer from a fixed list. It returns the answer, a probability for each option and a confidence number. It costs about $0.042 per million input tokens.

For each claim, Jev picks one of seven findings, and the router rule (below) turns that finding into a route:

Jev and Laya: what each is built for

Two typed-decision models are on the market. This research tested Jev only. Everything below about Laya comes from its maker's published documentation and comparison page, checked on 2026-09-26, not from a Yardstick measurement.

The two makers' own benchmark table shows where the designs part:

In plain terms: Laya is built for many fast, cheap decisions among a few options on short inputs, and can be run on your own hardware; Jev is built for decisions among many options on long inputs, with better-calibrated probabilities, at the cost of a hosted API and a slower round trip.

The task in this research is a seven-option choice on one claim and one short evidence window, in English. That is inside Laya's documented range as well as Jev's, so a like-for-like run of Laya on the same claims and the same accept rule would be the way to compare them. It has not been done, and nothing in this article's findings speaks to Laya's behaviour.

The rule under test

A claim is auto-accepted only under the rule in the table above: none at confidence of at least 0.8 on both of two repeat calls. This rule was fixed before any of the live runs and was not tuned on their results.

The evidence Jev sees is a window of text cut from the saved copy of the cited page, and it is not what the phrase suggests. Of the 590 live claims sent to Jev, 583 got a window:

So for all but one claim in twenty, "the evidence" is a passage a keyword heuristic guessed was relevant, not a passage anyone put forward in support of the claim. What that is worth is measured in Finding 4.

Two ways of using Jev were compared throughout:

Finding 1. Testing Jev as a router on live claims

What was tested. Freshly drafted dossiers were run through the production pipeline without publishing, and Jev ran beside it on every checkable claim. A claim is checkable when it carries a sourced tag and cites a page saved in the bundle. Fresh Claude Opus reviewers then judged the claims blind: they never saw what Jev, the code check of citations and tags or the whole-dossier LLM validator said. Two runs were reviewed:

What happened. A claim is bad when the blind reviewer, a Claude Opus agent that saw the claim, the saved copy of its cited page and the scoring rubric but never any model's output, gave it one of the three failing findings (a major score disagreement, a fact the page contradicts, or a claim the page does not support) or recorded that the claim is not true according to any public source; every other claim is good. The interval in the table is for the share of the accepted claims that are bad.

In the pilot, 9 of the 110 claims were judged bad by the blind reviewer, and Jev accepted none of them, at every confidence threshold from 0.5 to 0.95. In the scale-up, one of the 111 accepted claims carried a minor "unsourced" flag, which does not fail a claim.

An older test points the same way. On a historical set of 372 labelled claims, Jev's routing rule accepted none of the 120 recorded fails (95% upper bound 3.1%), on development and held-out vendors alike. [Note: 5 labels in that answer key were disputed and were reviewed on 2026-09-22; all 5 were ruled not fails, which is why the count is 120 rather than 125. The zero is unchanged, because Jev accepted none of the recorded fails either way.]

What it means. On this evidence, the claims Jev accepts are rarely bad: the point estimate is zero, and the 95% upper bound is about 3 in 100, somewhat wider in truth because claims cluster by vendor. That is the property a router needs.

Caveats.

• Zero bad out of 127 is not zero risk. The upper bound is about 2.9%, and it assumes claims are independent. They are not: three vendors supplied 33 of the 111 accepts in the scale-up, so the true uncertainty is somewhat wider.
• The answer key is Claude Opus, checked only in part by a person. Two Opus reviewers agreed on 29 of 30 re-judged claims in each run (kappa 0.90 in the pilot, 0.65 in the scale-up; kappa is low in the scale-up because bad claims are rare there, so one disagreement weighs heavily). The founder blind-labelled 40 of 60 pilot claims drawn at random for a human check. Before the comparison, the 13 score rows were set aside, because Jev does not judge them and the two score options had not been used consistently. One sampled claim that was only a sentence fragment was set aside as well. On the 26 fact claims left, the founder and Opus gave the same verdict on 24 (92%, 95% interval 76% to 98%; 23 of 26 if the one claim where the two Opus reviewers split is counted as a fail), including all 9 that Jev auto-accepted. Each called exactly one claim bad, and not the same one: both were claims with a detail the cited page did not state, which one reader treated as minor and the other as a fail, and kappa is about zero for that reason. On this sample, the founder and the answer key largely agree on which claims are fine; whether a person agrees on which claims are bad is not yet measured, and "bad" in this article still means "judged bad by Claude Opus".
• One Jev version (typesafe/jev-1.13-20260917), one route, all live calls within about eight hours on 21 and 22 September 2026 (UTC). The historical set's Jev outputs did not record which version served them.
• The drafts were re-drafts of vendors from evidence bundles frozen in June 2026, not a random sample of published claims.

What next. Keep measuring. A fixed set of 50 claims will be re-run every week to detect a silent model change, and a sample of accepted claims will keep going to review.

Finding 2. Testing how many claims Jev can route past review

What was tested. How much review work the router removes, and whether a better rule could remove more.

What happened.

• Every accept was a fact row: headline numbers (59), identity facts such as founding year or leadership (46), integrations (5) and one right-of-reply note. Score rows are rubric judgments, and Jev was not given the rubric.
• The evidence form mattered. Shown only the quote the draft itself gave, Jev accepted 2 of 110 pilot claims, because 103 of the 110 claims carried no quote at all. Shown a window from the saved page, it accepted 16.
• Other scores built from Jev's probabilities did no better. On the historical development set (the portion of the historical set of earlier-labelled claims that was used to develop the rule, split from the held-out portion by vendor), eight alternative scores were tested with 5-fold cross-validation grouped by vendor, each with its confidence threshold set so that it accepted no recorded fail on the training folds. The gap between the top two is too small to read:

• Jev's confidence is lower than its accuracy. When it answered none, 96% of those claims were recorded passes, while its average stated confidence was 70%. It is underconfident, not overconfident.

What it means. The router removes roughly a quarter of checkable claims from review, all of them factual statements. Jev could accept more claims if its confidence threshold were lowered, but on the historical set every lower threshold accepts recorded fails (the historical set's labels, not the blind reviewer's). At the current threshold, the rule accepts 25% of the recorded passes and accepts no recorded fail. Lower the threshold until it accepts about 65% of the recorded passes, and it also accepts about 6% of the recorded fails (5 of 85). That is what the chart shows: each lower threshold accepts more recorded passes and more recorded fails together. Where to set the threshold is a policy choice, not a tuning exercise.

Caveats. The accept share varies by vendor and section; three vendors gave 33 of the 111 accepts. In the scale-up, 105 of the 111 accepts were shown keyword-matched windows rather than a located quote, because the drafts rarely quoted their sources, and those windows are often a weak match: across all 551 keyword windows in the two live runs the overlap score runs from 1 to 28 with a median of 4, and 194 of them (35%) score 3 or less, low enough that the window may not be about the claim at all. Once drafts quote their sources, the evidence Jev sees changes, and these numbers need re-measuring. The historical set was built with a fail share (32%) far above what live drafts show.

Finding 3. Testing Jev as a second checker beside the existing validator

What was tested. Whether Jev, read as a pass or fail checker, flags what the pipeline's LLM validator (Mistral Small 3.2 24B, run here one claim at a time rather than on the whole dossier) misses. Both judged the same 372 historical claims (the historical set: claims with verdicts recorded by earlier reviews), one claim at a time, twice each.

What happened.

• The gap between Jev's and the validator's misses is not reliable (exact McNemar p = 0.18). The two missed many of the same fails: the odds ratio is 6.1 (vendor-clustered 95% interval 2.2 to 19). Independent checkers would give about 1.
• On the live pilot, Jev as a checker flagged many good claims. With the saved-page window it flagged 5 of the 9 claims judged bad by the blind reviewer and 38 of 101 good ones.
• The router rule still sent those fails to review. All 32 recorded fails Jev missed as a checker were sent to review, 23 of them because Jev picked a minor flag: it noticed something and rated it non-blocking.

What it means. Two models reading the same claim and the same evidence tend to miss the same things. Half of the fails both missed are defects neither model can see from the claim and its evidence: a wrong source tag or a cited page the model never saw (7 of 14). A second model is not an independent check. Deterministic checks of citations and tags target these better.

Caveats. The labels are mostly Claude Opus judgments from May and June 2026. The claim sentences were reconstructed from audit trails, and about 60 claims carry paraphrased evidence. The validator was run one claim at a time here, not in its production whole-dossier form. The cause codes behind the fails both missed were assigned by one coder who had already seen the labels and model outputs, so they were not blind. The 5 labels that were disputed were reviewed on 2026-09-22 and ruled not fails, and this finding is computed on the 120 that remain. Removing the 4 further fails whose defect sits in a page the model never saw leaves an odds ratio of 5.3, interval 1.5 to 17.6 (exploratory), so the overlap does not rest on them either.

Finding 4. Testing Jev against known false claims

What was tested. How often live claims are bad (judged bad by the blind reviewer, as defined in Finding 1), and whether the pipeline's two existing claim checks (the grounding check, which is a code check of citations and tags, and the whole-dossier LLM validator, Mistral Small 3.2 24B) flag them.

What happened.

• In the pilot, 9 of 110 checkable claims were judged bad (8.2%, 95% interval 4.4 to 14.8%), all 9 among the claims Jev did not accept. In the scale-up, weighted up, the sample means about 30 bad claims among the 369 Jev did not accept, somewhere between about 10 and 79.
• The drafts almost never quoted their sources: 7 of 110 pilot claims and 25 of 480 scale-up claims carried a quote. The grounding check treats a claim with a saved URL and no quote as grounded, so it had nothing to compare.
• Because so few drafts quoted, the evidence the classifier was shown was mostly a passage chosen by word overlap, not a quote. Of the 583 live claims that reached Jev with a page window, 551 (95%) were judged on a window chosen by word overlap rather than on a located quote. Those overlap scores run from 1 to 28 with a median of 4; 194 of the 551 (35%) score 3 or less and 44 score 1. The scale is three points per number shared with the claim and one per shared content word, and the only rejection threshold is a score of 0. So a score of 1 means the chosen passage had exactly one four-letter word in common with the claim and nothing else, and it was still passed to the model as that claim's evidence.
• 35 of those 551 windows (6%) are not continuous text from the page at all. The routine that cuts them splits the page into sentences, discards every fragment of two characters or fewer, and rejoins what survives with a single space. Where a discarded fragment sat between two passages, the two are spliced into what reads as one continuous quotation. In one window the join dropped the string " 1 " (a carousel counter) and produced the line "For every nonprofit [a named charity] partnered with [the vendor's parent company]", a sentence that appears nowhere on the page. This is a defect in the harness rather than a model behaviour, and it was found by a person reading the evidence, not by any check in the pipeline.
• The LLM validator passed all 5 pilot drafts, including two that print a total score of 0 out of 100. It passed 21 of the 22 scale-up drafts.
• A separate controlled test planted 31 defects (4 made-up figures, 18 citations to a page outside the evidence, 9 wrong provenance tags) into 22 of 23 real drafts:

What it means. The bad claims are not among Jev's accepts. They are among the claims it leaves for review, and the claim-level checks that exist today do not flag them. Drafts that were stopped at the publish gate (the last code check before anything is published) were stopped by its rule checks on the score table and penalty rows, not for these claims. Two changes follow: every sourced claim must carry a quote from its cited page, and a code check must compare citations, tags and the quote with the saved page. Splitting the validator into one call per claim is not the fix.

A third change is about how the evidence window is chosen, and it applies to this study's instrument as much as to the pipeline. A window picked by word overlap is a guess at relevance, and a window spliced across a dropped fragment is text that was never on the page. Both were feeding the classifier whose accepts are reported above as containing no bad claim.

They also reach the answer key, which is the more uncomfortable finding. The protocol tells each adjudicator to search the whole saved page, and each records what they consulted. On that record, 120 of 319 labels (38%) were made from the window alone. None of them is a fail: all 20 fail-flagged labels consulted the saved page, and most also consulted the vendor evidence digest, a web search or the live URL, so no bad-claim verdict rests on a window.

Nine of the window-only passes sit on a spliced window, and seven of those are among the 127 accepts, so all nine were re-judged against their full saved pages under the same protocol, blind to the original verdict. The splice changed no verdict. In every case the two runs of text were separated on the page by a single discarded token, not by distant material: a carousel numeral, a language-switcher "EN", a stray full stop, a dropped partner name in a logo list, and on one page four zero-width spaces. The deciding sentence sat wholly inside one run each time. Eight kept the same flag. The ninth changed flag, from none to claim_overspecification_minor, which is still a pass: the claim reads a platform total of $40 billion off a page that does say "$40 billion raised for good", but the page is the vendor's post-acquisition site, carries the acquirer's name in its title, says "[Vendor] is now [Acquirer] Pro" about 700 characters below the window, and footers as the acquirer. The figure is the acquirer's company-wide fifteen-year total, not this product's. That is a whole-page finding, not a splice finding, and window-only review is exactly what misses it.

That pointed at a larger defect in the review record than the splice. For 89 of the 127 accepts, no reviewer who labelled the claim had recorded opening the page it cites (25 of the accepts had a second reviewer, and where either one opened the page the claim is counted as read). So all 89 were re-read: the seven above in the re-judging of spliced windows, and the other 82 by four fresh reviewers running on a different Claude model (Fable 5.1; the first pass ran on Opus 5), blind to the earlier verdict, under the same rules plus two: opening the whole saved page was mandatory, and every label had to say whose page it was. None of the 82 changed verdict or flag; 80 were labelled at high confidence, and one was judged from the live page because the saved copy lacked the banner the claim came from. Across all 89, no verdict and no flag changed (the one changed flag in the re-judging of spliced windows was on a claim Jev had not accepted). Every one of the 127 accepts now rests on at least one recorded whole-page read, and the zero stands.

Both defects have to be fixed before Jev runs beside the pipeline in production. An excerpt must be continuous page text or mark its own elisions; a window that matches the claim weakly should be passed to the model as no evidence rather than as evidence; and an adjudication protocol that tells reviewers to open the page should verify that they did, rather than record it and never read the record.

Caveats. Small samples: 9 bad claims in the pilot and 3 in the scale-up sample. The planted defects are one kind of defect and may be easier or harder than real ones. One validator model with one prompt. The answer key is Claude Opus.

Finding 5. Testing whether a claim the page supports is a true claim

What was tested. The blind reviewers answered two separate questions for each claim: does the cited page support it as of its date, and, from any public source, is it true?

What happened.

Anonymized examples of the kinds of error found:

• A headquarters city taken from the dateline of a 2021 press release that never states the headquarters. Newer releases give a different city. (Pilot Vendor 4)
• A person named as CEO who, according to other sources, moved to a board role years earlier. The cited homepage does not name them. (Pilot Vendor 3)
• A funding total that added two rounds and left out an earlier round listed on the same page. (Pilot Vendor 3)
• A score rationale saying a product serves only one type of customer, while the cited About page lists others. (Pilot Vendor 2)
• Integration partners that do appear on the vendor's integrations page, cited instead to a blog post that names only part of the list. True, but not shown by the source. (Pilot Vendor 5)
• A claim that a vendor has no integrations, while its cited page lists many. The rubric does not define that field, and both reviewers who saw it marked their judgment at medium or low confidence. (Scale-up Vendor 12)

What it means. Two different properties are being checked, and a pipeline needs both. Source support can be checked against the saved page, by code and by models. Truth today needs fresh retrieval and dates. A claim that passes a source-support check can still be out of date, so every claim needs an as-of date and a refresh cycle.

Caveats. The "true per web" judgment came from the same Opus reviewers, with web access, and was not checked for every claim; it is a secondary field and never set the verdict. The pages were frozen in June 2026 and the claims were judged in September, which makes stale facts more likely than in a same-day draft. The historical cause codes were not assigned blind (see Finding 3).

Finding 6. Testing Jev against prompt injection

What was tested. Vendor web pages are written by the vendor, so evidence is text an outsider controls. That makes prompt injection possible: text on the page that gives the model instructions instead of facts. The test planted a sentence addressed to the reviewer, telling it the claim had been checked and to pass it, into the evidence window Jev reads. 39 recorded fails and 41 recorded passes from the historical development set (the portion of the earlier-labelled historical claims used to develop the rule) were run under four styles of planted text: a plain instruction, a fake editorial stamp, an HTML comment, and a note saying the evidence is outdated and the vendor confirmed the claim. The working sentences are not quoted here.

What happened.

• Jev's accept rule accepted none of the 156 planted fail cases (39 fails under each of four styles; 95% upper bound 2.4%).
• The planted text made Jev accept fewer claims, not more. Of the 41 recorded passes, it accepted 9 on evidence with no planted text, 10 with a control sentence with no instruction added, and 7, 3, 4 and 0 under the plain instruction, the fake stamp, the HTML comment and the outdated-evidence note (exploratory).

Real vendor pages already carry text aimed at AI readers. A scan of 1,894 saved source pages found one vendor whose pages address AI systems directly, with 21 matches across its saved pages, and another whose pages carry setup instructions for AI assistants. Three pages, two of them saved twice, carried 11 matches for invisible characters between them (8, 2 and 1). Jev reads windows cut directly from the saved pages, so this is the kind of text that can reach its input.

What it means. Under these strings, Jev accepted no planted fail case. When the text changed its answer at all, it changed toward sending the claim to review, which is the safe direction for a router. That is a property of the strict accept rule as much as of the model: under a rule that accepts few claims to begin with, planted text has to move Jev's confidence a long way before a claim is accepted. Evidence should still be screened in code before any model reads it, and a model's pass should never be the only check on a claim.

Caveats. Planted defects, fixed non-adaptive strings, one placement each, and one Jev version at one point in time. A motivated attacker could write a stronger sentence. The result shows that Jev did not accept a planted fail case under these strings, not that Jev cannot be made to accept one. The pass-side counts are small (41 claims per style) and exploratory.

Other results

• Hosting did not change verdicts. The whole-dossier LLM validator's model (Mistral Small 3.2 24B) gave the same pass or fail verdict on all 60 test claims whether pinned to DeepInfra, Parasail or Venice, or left on default routing, 12 calls per claim (0 of 60 differed; 95% upper bound 6%). Only its finer seven-option label moved, on 7 claims.
• Jev was stable across repeats. Three repeats on the same 60 claims gave 0 route changes (0 of 60, 95% upper bound 6%). Its confidence number did vary, by up to 0.19, on 50 of the 60 claims. The accept rule requires both repeats to agree.
• The blind reviewer mostly agreed with the drafts' scores. On 47 score rows with a stated number, the blind reviewer gave the same raw score 39 times and one step off 8 times, never two or more steps off (0 of 47, 95% upper bound 7.6%).

Caveats. These are small checks: 60 claims in each of the hosting and repeat tests and 47 score rows, one validator model and one Jev version. A zero here is a zero in this sample, not a zero rate. The score check's reviewer is Claude Opus, not a person.

What it all means

1. A cheap typed-choice classifier can remove part of claim review: on this evidence, about a quarter of checkable claims, all factual, with none of its 127 accepted claims judged bad by the blind reviewer and an upper bound near 3%.
2. It should be used as a router and not as a second checker. The recorded fails it misses as a checker are largely the ones the pipeline's existing whole-dossier LLM validator misses too.
3. The bad claims are in the pipeline's own drafts. Drafts rarely quote their sources, and the claim-level checks flagged none of the bad claims found. Quotes from the cited page and code-based citation checks come before any model-based check.
4. Source support and truth are different tests. Both are needed.
5. Evidence is untrusted input. Under planted text Jev accepted no planted fail case, and the text made it send more claims to review, not fewer. Evidence should still be screened in code before any model reads it.

What happens next

• Every sourced claim will carry a quote from its cited page, and a code check will verify citations and tags against the saved page.
• The saved-page windows Jev reads will be screened for instruction-like text in code before Jev sees them.
• Jev will run in shadow mode, logging its routes without changing any, then accept claims while a 10% audit sample of its accepts still goes to review, and only relax further once an acceptable bound is agreed. A weekly check for a silent model change runs throughout. The steps and the switch-off rules are set out in an internal implementation plan.

Methods

Data and splits

• Historical set. 372 claims with verdicts recorded by earlier reviews. Five recorded fails were reviewed on 2026-09-22 and ruled not fails, leaving 120; every historical figure in this article is computed on that answer key. Most recorded verdicts are Claude Opus validator judgments from late May to early June 2026 (307 of 372 claims); the rest come from the deterministic check's passes and from correction notes.
• Live pilot. Vendors re-drafted through the production pipeline from evidence bundles frozen in June 2026, not published, chosen from bundles no other part of the research used.
• Live scale-up. Vendors run in a pre-registered order until Jev had accepted at least 110 claims. This departs from the frozen plan, which scaled the run in a tune half and a confirm half to pick and then report an operating point: on 2026-09-22 the founder approved a cheaper scale-up that instead measures the share of the fixed rule's accepted claims that the blind reviewer judges bad, so no operating point was tuned on these claims.

Planted tests.

The 80 were drawn from the same development pool, so the 2026-09-22 ruling reaches them: one of them is no longer a recorded fail, and the planted-text test's figures here are computed on that key.

Blind adjudication

Fresh Claude Opus agents, with no shared context, judged each claim from the claim text, the saved copy of the cited page, the scoring rubric and the evidence digest, with web access to re-check sources. They never saw any model's output or the result of any check, and packets were shuffled with a stated seed. Each claim got one of the same seven findings Jev uses; the verdict was derived from the finding by a script. A secondary field recorded whether the claim is true according to any public source. A claim is bad when its finding was one of the three failing ones or the secondary field recorded it as not true according to any public source; every other claim is good. In the pilot, three agents split the 110 claims and a fourth re-judged 30 of them; in the scale-up, four agents split the 148 claims and a fifth re-judged 30.

Statistics

• Proportions carry Wilson 95% intervals. A result of 0 out of n is reported with its upper bound.
• Claims from one vendor are not independent. Differences and ratios carry vendor-clustered bootstrap 95% intervals (2,000 resamples, seed stated in each report).
• Paired comparisons on the same claims or dossiers use the exact McNemar test. Where two primary comparisons were tested together, p-values were Holm-corrected.
• Agreement between reviewers is reported as raw agreement and Cohen's kappa.
• Each question had one primary metric. Everything else is labelled exploratory. Cells with fewer than 10 cases are flagged as too small to read.
• Each test was run as its own workstream: a separate agent with its own pre-registration. Every one wrote its pre-registration (questions, hypotheses, primary metric, exclusions, seeds) and recorded its hash before any model call or any look at outcome data. Changes after that are logged as deviations. The agents did not read each other's outputs.

Limitations

These apply to every result above, and each section repeats the ones that matter most for it.

• The answer key is a model. Every "bad", "fail" and "miss" means "as judged by Claude Opus". Opus agreed with a second Opus reviewer on 29 of 30 claims in each live run. A person's blind labels matched Opus on 24 of 26 fact claims, including all 9 Jev accepts among them, but the two picked different claims as bad (one each), so agreement with a person on bad claims is not yet measured.
• One classifier version and one route. typesafe/jev-1.13-20260917 through OpenRouter, every live call between 20:00 UTC on 21 September and 03:20 UTC on 22 September 2026. The model is reached through a floating alias, so a later version may behave differently.
• Small and clustered samples. 127 accepts, which came from 26 vendors (all 5 in the pilot and 21 of the 22 drafted in the scale-up); 12 bad live claims in total across the two reviewed samples. Wilson intervals assume independent claims and are narrower than the truth.
• The evidence window is chosen by a heuristic. For 551 of the 583 live claims given a page window (95%), what the classifier was shown was not a quote anyone put forward but the passage on the saved page with the highest word overlap with the claim, plus eight sentences either side; a third of those matched weakly (overlap score 3 or less). 35 were not continuous page text, because the text splitter discarded short fragments and rejoined the survivors. Whether Jev's record of no bad accepts reflects it reading the window carefully or largely disregarding it is not settled by these runs, and the comparison that would settle it (showing Jev only the draft's own quote instead of a window cut from the saved page, run on the pilot) gave it no evidence at all for 103 of 110 claims, so it does not separate the two.
• The first review pass did not always open the cited page. The protocol asks the reviewer to read the window and then search the whole saved page, and each label records what was consulted. 120 of the 319 label records list only the window, and for 89 of the 127 accepts no reviewer had recorded opening the page. All 89 were re-read blind against the whole page by fresh reviewers required to open it; none changed verdict or flag. Every accept now rests on a recorded whole-page read, and every fail always did. The whole-page re-read used Claude models too (Opus 5 for the seven accepts re-judged because their window was spliced, Fable 5.1 for the other 82), so it confirms that the verdicts survive the page, not that a person would agree with them. The founder's blind check covered 9 of the 127 accepts and judged all 9 fine.
• Frozen, re-drafted evidence. The live runs re-drafted vendors from pages saved in June 2026. Fresh retrieval stops at a step where a Claude session writes the evidence digest (the curator step), which the pipeline does not automate, so no newly retrieved vendor was drafted.
• Reconstructed historical claims. The historical claim sentences were rebuilt from audit trails, and about 60 carry paraphrased evidence.
• Planted defects and fixed attack strings stand in for real ones in the controlled tests.
• One validator model and one prompt for every statement about the existing validator.

Reproducibility

• Dates. Runs on 2026-09-21 and 2026-09-22. Evidence bundles for the live runs frozen between 2026-06-12 and 2026-06-22. Historical labels from late May to early June 2026.

Models as served.

• Settings. Jev accept rule: finding none, confidence at least 0.8, both of 2 repeats. Evidence: the draft's quote located in the saved page plus two sentences either side, or, when there is no quote or it cannot be found, the sentence with the highest word overlap with the claim plus eight sentences either side, capped at 4,000 characters. Overlap is scored as three points per number shared with the claim and one per shared content word (four letters or more, minus a stoplist); the only rejection threshold is a score of 0, which yields no evidence and forces an escalation. Measured over the 583 live claims given a window (the other 7 of the 590 matched nothing on the page and went to review): 32 located quotes, 551 keyword windows, of which 35 are spliced across a fragment the sentence splitter discarded.

Cost. Jev is cheap: 1,920 calls in the scale-up cost about $0.07, or about $0.00004 per call. The Opus review sessions are not included below.

• Pre-registrations, scripts and raw outputs are kept with hashes for every test. The scripts, pre-registrations and claim-level results are published at the release folder, with vendor names replaced by codes. Claim text and page evidence are left out, because they would identify the vendors.

Independence and disclosure

Yardstick Research ran this evaluation on its own. TypeSafe did not see this article, the data or the results before publication, and had no role in designing or running the tests; Jev was called through OpenRouter's public API.

Yardstick Research has no commercial relationship with TypeSafe or OpenRouter.

Vendor names are withheld from every failure example. Many recorded errors concern Yardstick's own drafting, tagging or rubric, not the vendor, and naming a vendor next to an error would misattribute it. Planted defects were edits made for testing and are never shown under a real vendor's name.

The release

The pre-registrations, the analysis scripts and the anonymized claim-level results are published under /research/jev-router/release/. The README maps each file to the finding it supports and says what was left out and why. If a number does not reproduce, tell us: hello@yardstickresearch.app.