Head to head
Same brief, same rubric, same scale. Every model on the task side by side, or just the ones you pick: dimension scores, how many of their claims held up, and which issues each review raised.
It measured what everyone argued about.
Claude Fable 5.1Counts the calls, misprices them.
Every model here ran under the same conditions. How the runs were set up →
The best evidence chain, and one number that undoes its plan
Sixteen models have now reviewed this repository. Fable builds the most thorough case for the microphone finding and the only object-level memory plan, and it reaches the per-frame SPI writes that only two other audits found. Its sibling run on the same client found all four decode-critical issues; this one finds two and a half, and prices the vocoder at a sixth of its cost.
From the Claude Fable 5.1 audit, 17 September 2026.
Weighted score
Out of 100, on the grade scale.
Earned and lost, by rubric slot
Each bar is a model’s 100-point grade split into the rubric’s weighted slots; filled is earned, empty is lost. The slots line up, so a gap in one bar against a full slot in another is exactly where the models part ways.
- 1Accuracy & evidence30 pts
- 2Coverage of decode problems25 pts
- 3Root cause & prioritisation15 pts
- 4Fix plan & acceptance gates15 pts
- 5Originality & attribution10 pts
- 6Clarity & calibration5 pts
Scores on each rubric dimension
One panel per model: its score on each rubric dimension and the weighted total over the grade zones, in its own colour, with every other model on this task as a grey dot for scale. Hover a row for every model’s score and the auditor’s reasoning.
- The panel’s model, joined down the dimensions
- Every other model on this task
Claude Opus 592 A
Claude Fable 5.179 C+
Scores and weighted points
| Dimension | Weight | Claude Opus 5 | Claude Fable 5.1 | GPT-6 Astra | GPT-5.6 Sol | HY4 Preview | GPT-5.6 Luna | UNIONALPHA | Grok 4.6 | Muse Spark 1.3 Contributor | DeepSeek V4.1 Flash | GLM 5.3 | Gemini 3.8 Flash | Qwen3.8 Max | Qwen3.8 Flash | GLM 5.3 Flash | DeepSeek V4 Pro | Difference |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Accuracy & evidence | 30% | 94 · 28.2 pts | 85 · 25.5 pts | 92 · 27.6 pts | 91 · 27.3 pts | 85 · 25.5 pts | 82 · 24.6 pts | 90 · 27.0 pts | 74 · 22.2 pts | 74 · 22.2 pts | 73 · 21.9 pts | 68 · 20.4 pts | 65 · 19.5 pts | 70 · 21.0 pts | 62 · 18.6 pts | 64 · 19.2 pts | 61 · 18.3 pts | +9 |
| Coverage of decode problems | 25% | 86 · 21.5 pts | 62 · 15.5 pts | 84 · 21.0 pts | 70 · 17.5 pts | 56 · 14.0 pts | 58 · 14.5 pts | 65 · 16.3 pts | 59 · 14.8 pts | 54 · 13.5 pts | 49 · 12.3 pts | 52 · 13.0 pts | 46 · 11.5 pts | 39 · 9.8 pts | 40 · 10.0 pts | 34 · 8.5 pts | 33 · 8.3 pts | +24 |
| Root cause & prioritisation | 15% | 96 · 14.4 pts | 84 · 12.6 pts | 88 · 13.2 pts | 84 · 12.6 pts | 80 · 12.0 pts | 80 · 12.0 pts | 60 · 9.0 pts | 70 · 10.5 pts | 72 · 10.8 pts | 66 · 9.9 pts | 74 · 11.1 pts | 74 · 11.1 pts | 71 · 10.7 pts | 68 · 10.2 pts | 62 · 9.3 pts | 64 · 9.6 pts | +12 |
| Fix plan & acceptance gates | 15% | 93 · 14.0 pts | 86 · 12.9 pts | 92 · 13.8 pts | 90 · 13.5 pts | 84 · 12.6 pts | 80 · 12.0 pts | 70 · 10.5 pts | 75 · 11.3 pts | 71 · 10.7 pts | 70 · 10.5 pts | 68 · 10.2 pts | 70 · 10.5 pts | 71 · 10.7 pts | 70 · 10.5 pts | 62 · 9.3 pts | 60 · 9.0 pts | +7 |
| Originality & attribution | 10% | 95 · 9.5 pts | 84 · 8.4 pts | 90 · 9.0 pts | 84 · 8.4 pts | 82 · 8.2 pts | 72 · 7.2 pts | 76 · 7.6 pts | 72 · 7.2 pts | 70 · 7.0 pts | 74 · 7.4 pts | 70 · 7.0 pts | 70 · 7.0 pts | 66 · 6.6 pts | 64 · 6.4 pts | 58 · 5.8 pts | 62 · 6.2 pts | +11 |
| Clarity & calibration | 5% | 92 · 4.6 pts | 88 · 4.4 pts | 86 · 4.3 pts | 86 · 4.3 pts | 88 · 4.4 pts | 84 · 4.2 pts | 80 · 4.0 pts | 78 · 3.9 pts | 80 · 4.0 pts | 74 · 3.7 pts | 72 · 3.6 pts | 84 · 4.2 pts | 78 · 3.9 pts | 70 · 3.5 pts | 68 · 3.4 pts | 48 · 2.4 pts | +4 |
| Weighted total | 100% | 92 · A | 79 · C+ | 89 · B | 84 · B | 77 · C+ | 75 · C+ | 74 · C | 70 · C | 68 · C− | 66 · C− | 65 · C− | 64 · D | 63 · D | 59 · D | 56 · D | 54 · D | +13 |
Every claim, checked
One cell per claim in each audit’s claim table, checked at the lines it cites. Hover a cell for the claim and the auditor’s note. The reviews made different numbers of claims, so the bar under each grid gives the shares.
Claude Opus 5’s claim table · Claude Fable 5.1’s claim table
- Holds
- Qualifiedoverstated, miscounted or doubtful
- Wrong
20 hold · 1 qualified · 0 wrong · 95% hold
15 hold · 4 qualified · 1 wrong · 75% hold
What each review raised
Claude Opus 5 found 4 of the 4 decode-critical issues; Claude Fable 5.1 found 2 of the 4 decode-critical issues. Every tracked issue below, with the reviews that were already in the repository for reference.
Issue signatures
Each review as a trace across the tracked issues. Where traces part is where the reviews differ; hover a column for every review’s entry. Column numbers match the table below.
- raised
- partly
- absent
- Decode-critical
| # | Issue | Project docsEarlier repo review · in the repository | Claude Opus 5Model · Sep 17 | Claude Fable 5.1Model · Sep 17 |
|---|---|---|---|---|
| 1 | Capture export stages the wrong I2S halfwords | absent | absent | absent |
| 2 | Capture parser certifies an incomplete stream | absent | absent | absent |
| 3 | pcm_starve never increments | absentdocumented as working | absent | absent |
| 4 | No static RAM margin | partlymargins still to measure | raisedwith the overlay buffers named | raisedwith the objects to reclaim named |
| 5 | The I2S stream is most likely microphone audio Decode-critical | partlyopen, leaning sceptical | raisedproved from 0xE0=0x89 and sound.c | raisedtraced to the buffer’s only consumer |
| 6 | HR-C6000 de-emphasis on the capture path Decode-critical | absent | raisedmeasured at 48.9% SER | raisedwith the 300–3400 Hz bandpass |
| 7 | AT1846S FM filters, low-frequency bit, 25 kHz Decode-critical | partly“require characterization” | raisedall three, with the DMR fix | raisedall three, FM against DMR |
| 8 | 0x10=0x6E hybrid state; 0x36 dual role | partlybring-up clock rules | partly0x36 and 0xE0, not 0x10 | raisedboth, as the per-frame writes |
| 9 | Manual: I2S frame clock “must be 8KHz” Decode-criticalmissed or ruled out | absent | raisedH2 | absent |
| 10 | One-layer 4FSK test mode as a P25 tap missed or ruled out | partlystock BER-test block only | raisedRoute A, with an acceptance test | absentruled out on DMR framing |
| 11 | ±10% health gate versus ±1% timing clamp | absent | absent | absent |
| 12 | Fail-closed muting at LDU cadence | absent | partlyvia the MFID path | partlylate entry mutes seven frames |
| 13 | Non-standard MFID mutes clear calls | absent | raisedM1, with the Motorola case | absent |
| 14 | Test waveform shares the receiver’s RRC filter | partly“synthetic RRC/AWGN” caveat | raisedtested, called harmless | partlyasks for a recorded waveform |
| 15 | MCU runs at 72 MHz | raised | raisedand measured against it | raisedwith the PLL arithmetic |
| 16 | Vocoder needs 11–16× the 72 MHz CPU Decode-criticalmissed or ruled out | partlydecode timing unmeasured | raisedmeasured 6.5× on the demo build | partly11,812 trig calls, priced at 60–120% |
| 17 | Direct discriminator tap (M17 mod) | raised | raisedRoute B, with the ADC channels checked | raisedwith DC coupling and the ADC plan |
| 18 | Phase 2 architecture and scope | partlynot implemented | raisedincluding the descrambler seed | raisedincluding the NET_STS_BCST seed |
| 19 | Two unverified SPI writes per decoded 20 ms frame | absent | absent | raised0x36 and 0x10=0x6E |
| 20 | Capture sessions lack epochs | absent | absent | absent |
| 21 | Ring and tick real-time budget | partlydeadlines unproven | raisedthe 21 ms ring against a 55–190 ms stall | partlyper-window MAC cost only |
| 22 | Clock config 3 assumes 12,288 Hz; the codec formula gives 12,000 | absent | absent | absent |
| 23 | Clock-config writes bypass the verified SPI writer | partlySPI retry note | partlycalls the heal machinery fragile | absentcalls the machinery retirable |
| 24 | Stock squelch re-arms FM audio (0x10=0x80) during monitoring | absentassumes it can’t re-arm | absent | absent |
| 25 | Stale clear-call state releases a new call’s first frames | absent | absent | absent |
| 26 | No frequency tracking; ad-hoc timing loop gains | partlya code comment calls the DC estimate biased | raisedDC fit once per window, with the fix | raisedwith the carrier-offset arithmetic |
| 27 | Unknown talkgroup opens audio (fail-open gating) | absent | absent | absent |
From the provenance table in the Claude Fable 5.1 audit (17 Sep 2026), the newest audit on this task. Shaded columns are reviews that were already in the repository.
How each run went
Descriptive, not graded, and in grade order rather than ranked: one run per model. Run time is wall-clock, so it includes the provider’s speed, rate-limit waits and retries, and token counts vary with each model’s tokenizer. Every run followed Clean Room 1.0, which is what makes them comparable.
| Model | Run time | Agent steps | Tool calls | Output tokens | Route | Launch |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 25 min | 120 | 54 | 185K | Claude Code CLI, subscription | one-shot |
| Claude Fable 5.1 | 12 min | 71 | 44 | 249K | Claude Code CLI, subscription | one-shot |
Every run figure
| Model | Started (UTC) | Output tokens | Reasoning tokens | Context tokens | Messages | Sessions | Retry budget |
|---|---|---|---|---|---|---|---|
| Claude Opus 5 | 18 Sep 2026, 02:44 | 185,228 | 118,897 | 17,469,122 | – | 1 | – |
| Claude Fable 5.1 | 18 Sep 2026, 03:10 | 249,363 | 167,839 | 8,687,277 | – | 1 | – |
Context tokens are input plus cache reads and writes: mostly the conversation re-sent at every step, so they grow with steps × context size. The retry budget is how many API retries the harness allowed.