Every gate agreed. And every gate was wrong

This is one of the most useful failures I have on record, because it is exactly the situation that produces a dashboard full of green lights on top of a system full of errors.

The setup

The probe was measuring a relative ordering: two tiers of the same vendor's model family, a strong reasoning model and its cheap small sibling, across a ladder of increasingly harder requests. Each item scrambled a set of names into a transitive chain and asked how many names fall strictly between two of them; in this first run the ladder climbed from 12 names to 24. The correct answer is a count that never appears in the prompt. To defeat the obvious shortcut the items were built so that the naive positional count differs from the true answer. The expected ordering was that the strong model does at least as well as the small one.

Around the pipeline sat four automated checks, each aimed at a specific, real failure mode. A balance check (φ-balance) confirmed the design was symmetric. A gold-correctness check confirmed each item's stored answer really was the strict between-count, and that the positional shortcut would not pass. A scoring rule (contains_token) decided correctness by looking for the gold answer in the response. A floor-health check confirmed no cell had collapsed to a degenerate distribution.

These were not decorative. Days earlier the same family of checks had caught a real false failure: a response of $44 scored wrong against a gold of $44.00 and the floor check had already stopped a different probe from running at all when its floor came back saturated. They had a track record.

What happened

The probe reported this:

Read naively: the small cheap model is flawless while the flagship reasoning model falls apart as the task gets harder. A coherent, mechanistic, publishable-looking story. It is the exact inversion of the truth. Had it been recorded, the paper would have claimed that the cheap model outperforms the flagship on hard multi-hop reasoning. Every check came back green.

φ-balance passed because the design was balanced; the inversion was in the measurement, not the layout. gold-correctness passed because the gold answers were genuinely, verifiably correct and the probe reconstructed every item's ordering and confirmed each stored answer against it. Nothing was wrong with the answer key. contains_token returned a clean score with no error, because the gold string was, in the most literal sense, present in the response. floor-health passed because no cell was empty; every cell was full, and every cell was full of the wrong thing.

What the transcripts showed was two artifacts pointing in opposite directions.

The strong model produced false negatives. Its chain of thought consumed the whole fixed 2048-token budget, so the answer was truncated before it was ever emitted — one response ended as a 279-character fragment. Longer chain, more truncation, and the tidy monotone decline from 0.93 to 0.23 was not capability at all. It was the budget.

The small model produced false positives. It enumerated all 23 pairwise relations as a numbered list: 1. …, 18. Rosa > Silas, 19. …. The gold answers were counts in the same range, 14, 16, 18. contains_token matched the list index, not the answer. A coincidence, scored as correctness, at every single rung.

The insidious part? Two errors in opposite directions combined into one self-consistent narrative. It does not look broken, it presents itself as a finding.

The reason none of the checks fired is that every one of them checks a property of the artifact: is it balanced, is the answer key right, is the score non-degenerate, is the gold string present. Not one of them checks the thing a person checks by reflex, which is whether the number makes sense given what one already knows about the world. And the four were not four independent views of the same claim. Two were corpus-construction checks that had been satisfied before this probe ever ran, one was the scoring rule that was itself the bug, and one looked at the shape of the score distribution rather than its truth. Four green lights, no coverage.

What caught it

The small model was perfect at every rung. Nothing is perfect at every rung, and a cheap model does not beat a flagship on multi-hop reasoning. I saw the column of 1.00s, did not believe it, and pulled the thread by dumping the raw transcripts, which took about seventeen minutes and showed immediately that both arms were artifacted.

The correction was not a one-line fix, and this is the part I would want a reader to take away. It had two halves, one for each artifact. The output budget went from 2048 tokens to 8192, which gave the strong model room to finish thinking and answer. And the scoring rule was replaced: structured-answer extraction against an explicit answer marker, with three outcomes instead of two: correct, incorrect, and unparseable, where unparseable never collapses into incorrect. Around it went a validity gate that voids a rung entirely if either arm exceeds 5% unparseable, detection for the two degenerate modes (hitting the token ceiling, and repetition loops), and a pre-check confirming both models emit the answer marker cleanly on trivia, so the new instruction could not itself be the thing that moved the result.

Splitting "wrong" from "did not emit an answer" is what dissolved both artifacts at once: the strong model's truncations stopped being scored as wrong answers, and the small model's list indices stopped being scored as right ones.

The new scoring contract was written down and pre-registered before the re-run, so it could not be tuned until it produced a congenial answer. The whole cycle was one day: probe at 13:20, transcripts read and both arms proven artifacted by 13:37, scoring contract pre-registered at 16:06, sound re-measurement and result sealed by 23:49.

The sound re-run also extended the ladder to 32 and 40 names, six rungs in all. On the repaired instrument the strong model is perfect at every length from 12 to 40 names, with zero unparseable responses anywhere on the ladder. With the larger budget it no longer truncates at all, its entire apparent collapse had been the 2048-token ceiling, and nothing else. The small model is 0.89 to 1.00 correct on the responses it actually finishes.

Which leaves a genuinely confusing-sounding result, so it is worth being explicit: both models truncate in this story, at different points in it. Before the fix, the truncation that mattered was the strong model's, and it manufactured the fake collapse. After the fix, the strong model's truncation is gone entirely and what remains is the small model's own failure to finish. Its unparseable rate climbs with length to 40% at 40 names, roughly half truncation and half dropped answer markers and repetition loops. Two different arms, two different runs, either side of the budget raise.

So the truth was that there is no capability gap here at all. The two tiers reason equivalently on this task, and the only real difference is that the small one, at extreme length, fails to emit an answer rather than getting it wrong. On the responses it completes it is still right. The artifact had manufactured a gap and then pointed it the wrong way.

And the honest instrument reported less. Under the validity gate only two of the six rungs were admissible at all, and the headline went from a clean story to a null result.

Why I preserved it

It is a clean instance of a general property. An automated check can only fail an artifact on a dimension it wasn't built to inspect, and often the dimension requiring inspection is the one nobody anticipated. Adding gates does not close this, because each new gate is one more structural property, and the error that gets through is by definition the one no structural property caught. Worse, the checks are themselves a producer, with their own assumptions: here the scoring rule and the checks around it shared a blind spot, which is why agreement between them carried no information. When something does catch an error like this, that something is almost always outside the system that produced the artifact: a person, or an independent check that does not share the producer's assumptions.

This is not an argument against automated gates. They caught real problems before this incident and after it. It is an argument against reading "all gates green" as "the result is trustworthy." Green means no known structural failure mode fired. It does not mean the claim is true, and the gap between those two is where the expensive errors live.

A closing point: the three-outcome contract that came out of this incident later caught something on its own, on a different model family: a formatting mismatch that the old two-outcome scoring would have recorded as a batch of wrong answers. Under the old rule it would have flipped to the opposite conclusion and triggered a whole round of spending to chase it. The gate built out of one human catch went on to prevent the same class of error without a human present. That is the actual argument for gates, not that they are sufficient, but that a gate designed from a real caught error is worth far more than a gate designed from imagination.

If you build evaluation pipelines, the useful question is not how many gates you have. It is whether any part of your process is allowed to distrust a plausible-looking number and hold the line: an independent evaluation with the necessary information and the authority to stop things. That could be code, a model, or a person. The type of evaluator matters less than the dimension being evaluated and the authority given to the output.

This is one incident, not a study. The four checks were not badly designed; each was right for what it checked, and the scoring rule that broke was sound in the regime it was built for: small gold values on short items, and became unsound when the items grew long and the gold values grew into the range of ordinary list indices. The claim is narrower and perhaps more general: the class of error that slips through a series of structural checks is systematically the class those checks were not built to see, and catching it took a source of doubt from outside the software's frame of reference.