By Daniel Huber · original experiment · October 10, 2026 · 7 min read
The short version
Gemma 4 12B, the model we used, builds each word it writes through a stack of 48 layers. A lens reads the state at one of the middle layers and translates it into the words the model is leaning towards, before the model has finished.
We took answers that Gemma 4 12B had written, changed one fact in each to make it wrong, and measured whether the model reacted. It did: after a planted error, the rest of the text surprised it more than after a harmless rewording, in 25 of 30 cases. Read through two lenses at the middle layer, the same text showed no such reaction; the lenses did no better than chance. Both lenses also failed a basic check of whether they reflect what the model actually computes.
What this means for anyone building agents: here, the model’s own surprise was the useful warning sign, and it costs nothing to measure. Reading the middle of the model added nothing, at least at this layer.
Verdict
When one claim in Gemma 4 12B's own answer is turned into an error, the model's predictions of the unchanged text that follows get worse in 25 of 30 cases: it registers the mistake. Read at layer 24, neither the J-lens nor the R-lens adds anything to that, and both fail the check of whether they describe the model's computation faithfully.
Original measurement20 of 30
Output, on the edited words
18 of 30
J-lens, on the edited words
10 of 30
R-lens, on the edited words
30 error/neutral pairs, one model, one layer · the output's 20 of 30 on the edited words is p = 0.099 against chance; 25 of 30 on the words after is p = 0.0003 · per-pair records not published
What a lens is
A lens is a translator from the model’s unfinished internal state into ordinary words.
A language model builds each next word through a stack of layers; Gemma 4 12B has 48. The output shows only the end of that computation. A lens reads one of the intermediate states and translates it into words. The simplest, the logit lens, applies the model’s own output layer (the final step that turns a state into word probabilities) to a middle state. A Jacobian lens (J-lens) is fitted instead from how the model’s output responds to small changes in that state, averaged over many prompts (in mathematics, that response is called the Jacobian); this project used the public implementation. The R-lens is a published variant that changes how that response is computed, to make the readout more faithful on early layers.
The hope behind the project: if a wrong claim takes shape inside the model before it reaches the output, a lens might show it earlier or more clearly than the output’s own confidence. That would give an agent a warning signal before it acts.
Stable is not faithful
A lens is only useful if it gives the same answer when built twice, and if its answer matches what the model really does. The R-lens passed the first check; neither lens passed the second.
Both lenses were fitted for layer 24 on 744 prompts from a frozen corpus of 5.1 million tokens, then checked against two pre-registered bars.
| Lens | Stability (bar 0.90) | Faithfulness (bar 0.5) | Logit lens, same check |
|---|---|---|---|
| J-lens | 0.579 | 0.159 | 0.020 |
| R-lens | 0.939 | 0.098 | 0.016 |
Stability asks whether two lenses fitted on two halves of the prompts agree: 0.939 means that, of the ten words each reads out at a position, 9.4 on average are the same. Faithfulness nudges the layer-24 state in the directions the lens picks out and correlates the change the lens predicts in the output with the change that actually happens. A correlation of 1 would mean the lens predicts the change perfectly, 0 that it predicts nothing; the bar is a median of 0.5.
The R-lens is far more stable and even less faithful. Part of its stability comes by construction: its modified computation makes the fitted map depend less on the individual prompt. Both lenses beat the logit lens on faithfulness, the J-lens by a factor of eight, but neither comes close to describing what the model does. That alone failed the project’s own gate. Its plan listed what could follow: stop, fit other layers, or test for errors anyway as a recorded deviation. The planted-error tests below took the third option.
The planted-error test
This is the main test: change one fact in an answer and see which readout notices.
The first version paired a correct reference answer with an answer containing a planted error, on questions about financial products. It was not a valid test: the error answers were written as fluent prose while the references were terse notes, and at the first point where the two differed, the model preferred the error answer’s wording in 17 of 19 pairs. Its output caught only 8 of 19 errors. The test measured style, not error-noticing.
The second version used the model’s own answers. In each of 30 pairs, one correct claim in an answer to a development question is changed in two ways, both into an error (a flipped sign, a self-contradiction, a fabricated fact, a wrong attribution) and, as a control, into a neutral rewording at the same spot; the rest of the text is identical. A readout catches a pair when the error version surprises it more than the neutral one. A readout that noticed nothing would still catch about half the pairs by luck, so chance is 15 of 30.
| Row | On the edited words | In the 32 words after | Difference |
|---|---|---|---|
| Model output | 20.0 | 25.0 | +5.0 |
| J-lens, layer 24 | 18.0 | 15.0 | −3.0 |
| R-lens, layer 24 | 10.0 | 12.0 | +2.0 |
| Logit lens, layer 24 | 14.0 | 13.0 | −1.0 |
On the edited words, the output caught 20, the J-lens 18 and the R-lens 10, significantly fewer than the output (p = 0.013: if there were no real difference, a gap this large would turn up only about once in 77 tries). The clearer signal is in the 32 identical words after the edit: after an error, the model’s own predictions of those words got worse in 25 of 30 pairs. The false claim disturbs what comes next. None of the layer-24 readouts shows this: 15, 12 and 13 of 30, all at chance. Under the pre-registered rule, the verdict is “no evidence the lens adds anything”.
What this does not rule out
- Other layers. Error-noticing could form after layer 24 and before the output, or in the early layers the R-lens was designed for. Only layer 24 was fitted.
- A trained probe. Both lenses read the state through a fixed mapping onto words. A probe (a small model trained to read a layer directly for one question, here “is this wrong?”) on layer 24 could find information they discard.
- Real failures. Planted errors test whether the model notices an error in text it reads, not whether it can predict its own mistakes.
For agent builders, the useful signal here needed no interpretability tooling: how much more surprised the model is by its own text after a claim than after a neutral rewording. That is cheap to measure, and worth trying before reaching for internal readouts.
Method
How the lenses were built and tested, and the limits of the evidence.
The model was Gemma 4 12B, instruction-tuned, in bf16 (16-bit numbers, the usual precision for running such models), run on one GPU workstation (an NVIDIA DGX Spark-class machine). Both lenses were fitted for layer 24 of 48 on 744 prompts from a frozen corpus of 4,091 documents: general web text, encyclopedia articles, books, public filings and regulatory texts, and model chat turns. The R-lens fit took 16.1 days. A larger, pre-registered series of fits would have taken about 85 days and was not run.
The questions are a private set about financial products, split by question. Only development questions (the ones set aside for building the test) were used; the held-out questions and the question texts are not published. The 30 edits were drafted by the coding agent and approved in bulk by the experimenter, not written independently.
Each step’s design and pass rule was committed before its run, in a private repository; the first planted-error design was committed minutes before its run. Fits are not bit-reproducible at 512 tokens. The per-pair records stay on the GPU machine; the p-values on this page were recomputed from the counts.