Emission evals

The product claim behind every auri catalog: language models emit these vocabularies cleanly, cold. This page is the evidence — the same harness that gates our releases, its results published, one matrix per catalog. Last updated 2026-08-20.

ops

Scenario	GPT-5.6	Claude Fable	Claude Sonnet
`status-view`	✓	✓	—
`payments-snapshot`	✓	—	✓
`incident-hard-mode`	✓	✓	—
`deploy-live-ops`	✓	✓	—
`service-passport`	✓	—	✓
`weekly-review`	✓	—	✓
protocol · date	harness 2026-08-19	fresh-session 2026-08-17	fresh-session 2026-08-17

forms

Scenario	GPT-5.6	Claude Fable	Claude Sonnet
`contact-form`	✓	✓	—
`postmortem-intake`	✓	—	✓
`deploy-approval-form`	✓	✓	—
`settings-server-validation`	✓	—	✓
`survey-sections`	✓	✓	—
`signup-checks`	✓	—	✓
protocol · date	harness 2026-08-19	fresh-session 2026-08-19	fresh-session 2026-08-19

intel

Scenario	GPT-5.6	Claude Fable	Claude Sonnet
`surface-insight`	✓	✓	—
`account-velocity`	✓	—	✓
`risk-clusters`	✓	—	✓
`drill-path`	✓	✓	—
`case-audit`	✓	—	✓
`feedback-response`	✓	✓	—
protocol · date	harness 2026-08-20	fresh-session 2026-08-20	fresh-session 2026-08-20

✓ zero schema errors · ✕ failed validation · — not yet run for this model

What a pass means

The model receives the catalog's prompt-pack (ops · forms) as its system prompt — no other context, no retries, no examples beyond what the pack itself teaches — and one realistic scenario ("show the on-call engineer an incident view…"). Its entire output is validated line-by-line against the contract (ops · forms) with the same ajv validator our CI uses. A pass is zero schema errors across the full emission.

When these run

Two schedules, catching two different kinds of drift. Nightly, CI runs a smoke subset — the two widest-coverage scenarios per catalog — against a live model. That one exists to catch model drift: a provider ships a new checkpoint, a vocabulary that emitted cleanly last week stops, and we learn it from a red build rather than from a user. On every contract change, the full suite runs for each catalog whose contract was touched — that catches our drift, and it is the contract-first invariant enforced by machine instead of by discipline.

The matrices above are transcribed from full verified runs on the dates shown, not from the nightly smoke job. When a nightly run fails, the fix is a contract fix — never a prompt patch.

Scores are first-attempt; alerts are confirmed. A pass is still a single cold emission with no retries — that is what the matrices report and what the product claim means. But a red build is only worth waking up for if it reproduces, so a failed scenario is re-run once cold, and only a failure that happens twice turns the build red. We measured why on 2026-08-22: emission failures cluster by run rather than by scenario — reordering two scenarios made a previously perfect one fail, and the reverse order passed both minutes later against an unchanged contract. That is provider-side variance, and an alarm that fires on it is an alarm you stop reading.

Two protocols

harness — automated: npm run eval in packages/ops or packages/forms sends the pack cold via the provider's API and scores the result. Reproducible by anyone with an API key.
fresh-session — the original gate protocol: the pack given to a fresh model session with nothing else, output scored with the same validator. Documented run-by-run in the gate logs (ops · forms), which also record every contract fix these runs produced — including forms' component batching rule, found by a model failing the same way three times in a row.

Run it yourself

cd packages/forms   # or packages/ops
OPENAI_API_KEY=... npm run eval -- --models openai:gpt-5.6

The shared harness lives at scripts/emission-eval.js — providers activate on env keys (Anthropic, OpenAI, Gemini), results include the raw transcripts, and the --pack/--contract/--scenarios-file flags point it at any catalog or composed vocabulary. Scores on this page are regenerated from real runs, never edited by hand.