# Hallucination rate per model combination

Each row = one (step-01 / step-02 / step-03) model combination, pooled over the
vendors it was run on. Rate = ungrounded claims / total claims. Models are named
(they are the comparison); vendors are anonymized as `cohort-g<gold>`.

| step-01 / step-02 / step-03 | vendors | claims | halluc | rate |
|---|--:|--:|--:|--:|
| gpt-5-mini / grok-4.3 / o4-mini | 15 | 383 | 0 | 0.00% |
| gpt-5-mini / grok-4.3 / llama-3.3-70b-instruct | 15 | 407 | 0 | 0.00% |
| gpt-5-mini / qwen3-235b-a22b-2507 / o4-mini | 14 | 435 | 1 | 0.23% |
| gpt-5-mini / grok-4.3 / mistral-small-3.2-24b-instruct | 30 | 804 | 3 | 0.37% |
| claude-sonnet-4-6 / claude-opus-4-8 / claude-opus-4-8 | 30 | 1376 | 6 | 0.44% |
| gpt-5-mini / deepseek-chat-v3.1 / o4-mini | 29 | 891 | 5 | 0.56% |
| gpt-5-mini / gemini-2.5-pro / mistral-small-3.2-24b-instruct | 29 | 948 | 6 | 0.63% |
| gpt-5-mini / gpt-5 / mistral-small-3.2-24b-instruct | 29 | 1822 | 12 | 0.66% |
| gpt-5-mini / mistral-small-3.2-24b-instruct / mistral-small-3.2-24b-instruct | 29 | 846 | 6 | 0.71% |
| gpt-5-mini / grok-4.3 / qwen3-235b-a22b-2507 | 6 | 140 | 1 | 0.71% |
| gpt-5-mini / qwen3-235b-a22b-2507 / mistral-small-3.2-24b-instruct | 29 | 759 | 9 | 1.19% |
| gpt-5-mini / qwen3-32b / mistral-small-3.2-24b-instruct | 29 | 629 | 8 | 1.27% |
| gpt-5-mini / gpt-5-mini / o4-mini | 2 | 155 | 2 | 1.29% |
| gpt-5-mini / o3 / mistral-small-3.2-24b-instruct | 15 | 386 | 5 | 1.30% |
| gpt-5-mini / gemma-3-27b-it / o4-mini | 8 | 73 | 1 | 1.37% |
| gpt-5-mini / deepseek-r1 / o4-mini | 6 | 139 | 2 | 1.44% |
| gpt-5-mini / llama-3.3-70b-instruct / o4-mini | 3 | 64 | 1 | 1.56% |
| gpt-5-mini / gpt-4o-mini / mistral-small-3.2-24b-instruct | 15 | 243 | 4 | 1.65% |
| gpt-5-mini / gemini-2.5-flash-lite / mistral-small-3.2-24b-instruct | 15 | 483 | 9 | 1.86% |
| gpt-5-mini / mistral-large-2512 / o4-mini | 5 | 164 | 4 | 2.44% |
| gpt-5-mini / llama-4-maverick / o4-mini | 2 | 42 | 2 | 4.76% |
| mistral-small-3.2-24b-instruct / grok-4.3 / mistral-small-3.2-24b-instruct | 3 | 76 | 4 | 5.26% |
| gpt-5-nano / grok-4.3 / mistral-small-3.2-24b-instruct | 3 | 73 | 4 | 5.48% |
| gpt-5-mini / llama-4-scout / mistral-small-3.2-24b-instruct | 15 | 177 | 10 | 5.65% |
| gpt-5-mini / o3-mini / mistral-small-3.2-24b-instruct | 15 | 397 | 30 | 7.56% |
| gpt-5-mini / o4-mini / mistral-small-3.2-24b-instruct | 15 | 495 | 39 | 7.88% |
| gpt-5-mini / command-r-08-2024 / mistral-small-3.2-24b-instruct | 3 | 12 | 1 | 8.33% |
| gpt-5-mini / gpt-5-nano / mistral-small-3.2-24b-instruct | 14 | 586 | 79 | 13.48% |
| gpt-5-mini / gpt-4.1-nano / mistral-small-3.2-24b-instruct | 15 | 451 | 82 | 18.18% |
| gpt-5-mini / gpt-4.1-mini / mistral-small-3.2-24b-instruct | 15 | 512 | 133 | 25.98% |
| gpt-5-mini / gemini-2.5-flash / o4-mini | 1 | 32 | 11 | 34.38% |