Six passes: four language-model, one deterministic, one speech

How language output earns trust

A model can produce elegant German analysis that is structurally wrong, pedagogically unhelpful, or subtly false in context. Similarity scores do not settle that, so acceptance criteria live outside the response.

The four passes divide along how much of their correctness can be stated mechanically. Segmentation and lemma finding are checked structurally — segmentation must reconstruct the source and preserve token order, lemma output must account for every expected occurrence. Translation and glossing carry expert-authored assertions instead, because the failure that matters is pedagogical rather than structural: an answer can satisfy every shape rule and still teach the reader the wrong thing.

Glossing: what “contextually right” actually means

A word whose meaning shifts in context cannot be glossed with the dictionary sense or with the translation’s word. The reader is looking at the German, so they need both: what the word says, and what it is doing here. The convention is dictionary sense, /here:/ contextual sense, and the tests assert that the marker is present exactly where it belongs — over-marking an ordinary word fails as loudly as under-marking a shifted one.

• Hey, Schnecke! — “Hey, babe!” Schnecke(f) is a snail, and here it is an endearment. “Snail” alone strands the reader; “babe” alone hides what the word on the page says. Accepted: snail, /here:/ babe.
• Ich hab keinen Bock — “I don’t feel like it.” Bock(m) is a billy goat. Harder than the last one, because the sense the reader needs lives in the whole idiom rather than the word, which gives a model every excuse to copy “feel like” out of the translation and drop the goat entirely.
• Er stand früh auf — “He got up early.” A different property, and a structural one: the surface verb is stand, but the lemma is aufstehen — the separable prefix sits at the end of the clause. A gloss keyed to stehen (“to stand”) is fluent, confident, and wrong.

The prompt states the rules that go with these: gloss the lemma and not the translation token, keep comparatives on the base form, use /here:/ only where the sense actually shifts, never collapse the polite Sie into she, etc.

Translation: two rules a benchmark would punish

The translation fixture flags phrases with the property under test, and both named properties exist because the reader is looking at the German while they read the English.

• Idioms must carry their literal reading. Er hatte die Arschkarte gezogen is accepted as “He had drawn the short straw: drawn the ass card”; mit seinem Latein am Ende as “at his wits’ end: at the end of his Latin”. A clean idiomatic rendering alone would score well on any translation benchmark and would strand a reader who can see that the sentence mentions Latin.
• Words English must add are marked as added. German drops verbs English requires: Sie muss ins Bett becomes “She must /go/ to bed”, Sie kann Klavier becomes “She can /play/ the piano”. The slashes mark what was inserted, so the translation does not quietly claim a word is present when it is not.

Neither rule improves a similarity score. Both exist because a translation beside the original has a different job from a translation that replaces it.

Running it repeatedly, and looking at the output

Fixtures decide what can be stated in advance. A harness decides whether the model does it reliably. It runs a pass against the configured production model, and can run the same pass repeatedly — the consistency mode executes a set number of repeats and writes a report per run, because a pass that is right four times in five is a different proposition from one that is right every time.

Each run records latency, input and output tokens, and computed cost, so a prompt change that improves quality and doubles spend is visible as both. Batch size is a parameter rather than an assumption, since it changes accuracy and price together. Real provider calls are opt-in and excluded from the default suite; they cost money and are run deliberately.

What fixtures cannot encode, I read. Stored responses, their prompts, and their costs are inspectable in Avo, which is where production output gets checked for the kind of quality problem that has no assertion — a gloss that is defensible but unhelpful, a translation that is accurate and badly pitched. Encode what can be encoded; look at the rest.

Fixtures are adversarial rather than numerous — one segmentation passage carries guillemets, an em dash, an abbreviation (z.B.), a decimal with a suffix (3.147-mal), an ellipsis, a parenthetical, and a separated verb, each breaking a different naive implementation.

What cannot collapse into a boolean stays with the expert. Models are pinned per pass, because the four differ in quality, latency, and prose requirements. There is no invented quality score here; numerical claims need a dated result.

What happens after a provider call succeeds

HTTP success only says the provider returned something. Three questions remain: can the application validate it, was it persisted exactly once, and is there enough evidence to publish it.

Language passes. Pass state, response evidence, usage, and cost are recorded in domain context. A valid prefix is preserved instead of regenerating a whole batch; a single phrase can be rerun; repair does not quietly advance the normal workflow frontier.

Audio. Here the economics bite, because the expensive failure usually happens after the provider has charged. Narration is generated in word-bounded segments, each claimed before the paid call. On success the system immediately records cost, model, voice, checksums, correlation identifiers, and timing evidence, attaches the audio, maps character timing into phrase windows, and runs an independent certifier: lexical correspondence, monotonic timing, valid duration, non-overlap, complete coverage. Only certified audio satisfies readiness.

If speech is good but alignment fails, salvage repairs it from stored timing evidence without buying the narration again. A blocking hole stops new paid calls beyond it; already-good later segments are reused once it is repaired.

When a person has to decide. Linguistic quality is invisible to an exception tracker — a phrase can be valid and still wrong for the passage. Reader feedback carries signed context identifying the chunk and phrase, resolving in the operator surface into the source, prompt and model evidence, current artifacts, and dependencies. Repair tools produce reviewable candidates and explicit apply actions rather than mutating published data in the background. I build that surface assuming the operator will not always be me.