Model review audits
Method

How a review audit works

bench doesn’t grade multiple-choice answers or a model’s own account of its work. A model reviews a real codebase against a real engineering question, and then its review is audited claim by claim, with its evidence rerun.

The process

From brief to grade

  1. One brief, one working tree

    Every model on a task gets the same question about the same code, documents and tooling: “Find why this firmware struggles to decode P25 Phase 1 and Phase 2 voice on the radio’s HR-C6000, and what to do about it.”

  2. The model writes its review

    Findings, evidence and a fix plan, in its own format. Stealth models are graded without the auditor knowing who made them. Anything a model was given up front, such as an earlier model’s findings, is disclosed with its grade.

  3. Every claim is checked where it points

    Each claim is checked at the lines it cites, against the source, the datasheets and manuals, and the reviews already in the repository. It holds (sometimes with omissions noted), is qualified as overstated, miscounted, miscited or doubtful, or is wrong.

  4. The evidence is rerun

    Test suites, harnesses and builds the review relied on are run again, to separate real findings from artifacts of how they were produced.

  5. What it missed is measured

    Issues the review left out are modelled, not just listed. For the P25 task that means driving the firmware’s own demodulator through modelled receive filters, and measuring the decoder’s CPU cost from emulated instruction counts.

  6. Graded on a published rubric

    Six weighted dimensions give a score out of 100 and a letter grade. The rubric, weights and scale are fixed for a task, so grades on the same task compare directly.

The rubric · P25 firmware review

Six dimensions, weighted for the question

Weights reflect the question asked: find what stops P25 decoding and say how to fix it. Accuracy carries the most weight because a wrong review does harm; coverage and root cause together outweigh it because an accurate review of the wrong things does not help.

  • 30%Accuracy & evidence
  • 25%Coverage of decode problems
  • 15%Root cause & prioritisation
  • 15%Fix plan & acceptance gates
  • 10%Originality & attribution
  • 5%Clarity & calibration
30% of the grade

Accuracy & evidence

Do its line references, test results, figures and reproductions hold up when checked and rerun?

25% of the grade

Coverage of decode problems

Does it find the problems that actually stop the radio decoding?

15% of the grade

Root cause & prioritisation

Does it name the real blockers and put them first?

15% of the grade

Fix plan & acceptance gates

Is the plan concrete, ordered and gated by measurable checks, and would following it avoid new failures?

10% of the grade

Originality & attribution

What does it add beyond the reviews already in the repository, and does it credit them?

5% of the grade

Clarity & calibration

Is it clear, and does its confidence match its evidence?

Grade scale

From score to letter

The weighted total maps to a letter. C covers 65 to 79 in minus, plain and plus steps; below 50 is an F.

GradeWeighted score
A ≥ 90
B 80–89
C+ 75–79
C 70–74
C− 65–69
D 50–64
F < 50
The task · Firmware & RF

P25 voice decoding on the DM-1701

A DM-1701 handheld running an OpenGD77 tree with an experimental P25 receiver: an AT1846S RF front end and the HR-C6000 codec feed a demodulator, framer and the mbelib IMBE vocoder on an STM32F405 clocked at 72 MHz. Each model reviews the same working tree (46eebda) and writes up what stops the radio decoding and how to fix it.

The brief“Find why this firmware struggles to decode P25 Phase 1 and Phase 2 voice on the radio’s HR-C6000, and what to do about it.”

DM-1701 · HR-C6000 · AT1846S · STM32F405 · P25

The four decode-critical issues the audits track

  • The analog-FM receive chain stays on

    For a P25 session the monitor leaves the radio in analog FM mode: AT1846S high- and low-pass filters on, low-frequency audio blocked, 25 kHz bandwidth, and HR-C6000 de-emphasis on. Modelled through the firmware’s own demodulator, a high-pass at 75 Hz or above, or de-emphasis alone, leaves 0 of 180 voice frames.

  • The I2S stream is most likely the microphone

    The decoder reads the HR-C6000 slot that OpenGD77 labels “Mic Audio”, and register 0xE0 routes the microphone into LineIn1.

  • The I2S frame clock “must be 8KHz”

    The HR-C6000 manual says so; the monitor runs it at 24 kHz.

  • The vocoder can’t run in real time

    Emulated instruction counts put mbelib’s IMBE synthesis at 11–16 times the radio’s 72 MHz CPU; a loud 20 ms frame costs about 430 ms.

Limits

  • Filter corners for the AT1846S and HR-C6000 are modelled, not measured; the sweeps show the conclusions hold across them.
  • CPU figures come from instruction counts in an emulator, not cycle-accurate timing on the radio.
  • No firmware was edited or flashed, and no radio was touched.
  • A model is not a measurement.