A lie detector for a language model can be a probe: a small classifier that reads the model’s activations. In earlier experiments, a small model trained against a probe learned to keep it quiet12. A new probe, trained on that model’s answers, caught only some of its lies3.
One kind of lie stayed hard to catch. We call it a spin lie: the answer mentions the awkward fact, but it still misleads. One trained model admitted “25% cost overruns resulting from equipment failures” at a factory. Then it added: “However, the remaining 75% is on track within budget.”
The new probe read only one layer of the model. In a study of larger models, probes that combined five layers did better4. So a defender has an obvious next move: a probe that reads every layer.
If a model learned to slip past a lie detector, does a probe that reads every layer catch the spin lies a one-layer probe misses?
TLDR; Not settled. Our megaprobe reads every word at every layer. On spin lies, it scored 0.68 on average over four trained models, against 0.73 for the one-layer probe. On this score, 1 is perfect and 0.5 is a coin flip. Two other probes that read every word at many layers did no better: 0.70 and 0.59. The 0.59 probe had stopped training too early. Trained four times longer, it gained about 0.07 and still stayed below the one-layer probe on average. All three also failed to beat the one-layer probe on all lies on the untrained model. With twice the labels, the megaprobe caught up with the one-layer probe but did not pass it. So we cannot tell if spin lies hide from every probe, or if far more labels are needed.
On average, no probe reaches the 0.80 line, and every other probe sits below the one-layer probe.
How the test works
We reused four small models (Qwen2.5-0.5B-Instruct) that we had trained against one probe or eight probes, and the untrained model. Each plays a company’s assistant in chats from DolusChat, a public deception dataset5. The assistant knows an awkward fact, and a user asks about it. A judge model (Gemini 3 Flash) had labelled 1,000 answers per model. An answer mentions the fact when it uses more than a quarter of the fact’s content words. A spin lie mentions the fact, but the judge calls it deceptive. A Claude agent read 50 spin lies. It found 14 that spin the fact, 28 that use its words but contradict it, and 8 that look truthful.
We compared four probes:
- One-layer probe: the earlier probe, on one middle layer (12 of 24).
- Layer-average probe: the public recipe of Taufeeque and colleagues6. After the answer, the model is asked “Did you respond truthfully?” and answers “Yes”. One probe per layer reads that word, and the score averages the layers that fit well.
- Megaprobe: our build. It reads every word at every layer, learns where to look, and adds up what it sees there.
- Published probe: a similar probe from Hollinsworth and colleagues7, at its code’s default size.
We also built a second megaprobe with separate code, as a check on our own.
Each probe trained on 400 labelled answers from the same model, and scored only chats it had not trained on. AUROC is the chance that a probe ranks a random lie above a random truthful answer. The truthful answers here also mention the fact. So a probe cannot win by spotting a missing fact.
We fixed the rule in advance. Yes needed the megaprobe to average 0.80 and beat the one-layer probe. No needed it to stay clearly below 0.80. No also needed one check: on the untrained model, the megaprobe must rank all lies better than the one-layer probe.
What we found
The megaprobe stayed clearly below 0.80, at 0.68 (95% interval 0.64 to 0.71). The one-layer probe beat it on every model, by 0.03 to 0.10. The layer-average probe scored 0.61, about the same as a classifier that reads only the words (0.62).
The megaprobe failed the check. On the untrained model, it ranked all lies at 0.81, against 0.85 for the one-layer probe. So, under our rule, the answer is not settled. Our second megaprobe tied the one-layer probe at 0.70. It also just missed the check (0.83 against 0.84). The published probe scored 0.59 on spin lies and failed the same check (0.75 against 0.85).
The published probe had not finished training. We gave it 10 passes over its training answers. After them, it still fit those answers poorly. We found this after publication and trained it for 40 passes. On spin lies, it then gained about 0.07 on average and tied the one-layer probe on one model. On all lies, it stayed below the one-layer probe on every model, and it still failed the check (0.77 against 0.84).
The one-layer probe partly detects a missing fact. Against all truthful answers, including those that omit the fact, its spin-lie score fell from 0.73 to 0.67. It scores truthful answers that omit the fact closer to lies.
Our prediction failed. On all lies, the megaprobe fell 0.05 to 0.14 short of its score on the untrained model, where we predicted at most 0.03.
Should you trust our results?
- We used one small model (0.5B), two training seeds per setup and one judge.
- The Claude agent agreed with the judge on 22 of 30 truthful answers. Label noise like this keeps every probe below 1.
- 400 labels is few for a megaprobe. The published probe was trained on about 153,000 examples.
After publication, we tested whether more labels help. We trained the one-layer probe and the megaprobe on 100, 200, 400 and all labelled answers, about 800 per model. On spin lies, the one-layer probe rose slowly, from 0.68 to 0.71. The megaprobe rose faster, from 0.59 to 0.70. So its deficit shrank from 0.09 to 0.01, which is a tie. A smaller megaprobe, which reads 32 directions per layer instead of 128, also tied at 0.71. No probe came near 0.80. These fits used one draw of labels per size, so differences under about 0.03 are noise.
So, does a probe that reads every layer catch the spin lies a one-layer probe misses? Not settled. With 400 labels, three probes that read many layers caught no more spin lies than the one-layer probe. None beat the one-layer probe on all lies on the untrained model, the check we fixed in advance. With twice the labels, the megaprobe caught up with the one-layer probe but did not pass it.
For me, a megaprobe is not a free upgrade. A defender with a few hundred labels should test it against a one-layer probe on the same labels first.
Details for experts
In short: One fold map (5 folds grouped by chat, seed 1020) is shared by every model and probe. In each fold, every probe fits on the same class-balanced draw of 400 labelled answers from the other four folds: 5 draws, 3 for the megaprobe. The untrained model has only about 180 truthful answers in four folds, so its draws take all of them. Each answer’s score is the mean over the draws that held it out. Intervals are paired bootstraps over chats (1,000 resamples).
- One-layer probe: logistic regression (C = 0.05, standardised inputs) on the layer-12 activation averaged over the answer’s tokens, as in the earlier post.
-
Layer-average probe: the readout of Goldowsky-Dill et al.8 as used in the Obfuscation Atlas (§4.3; code at obfuscation-atlas @12abf65,
detectors/train.py:581) and in FIBS (App. A.2). One linear probe per layer on the final follow-up token, inputs divided by the layer’s RMS norm. Layers with training cross-entropy above 0.6 are dropped, and the score is the mean of the kept layers’ sigmoids. Our first run under-fit every layer, so no layer passed and the code fell back to one layer (0.58 on spin lies). We report the refit with the public code’s solver, which kept almost every layer (0.61). Two other refits scored 0.54 and 0.59, so the recipe stays near 0.54 to 0.61. -
Megaprobe: in each layer, every token vector is projected on the top 128 principal components of the training tokens. One softmax then runs over all (token, layer) pairs:
\[\text{score} = \sum_{t,l} \operatorname{softmax}_{t,l}(q_l \cdot z_{tl} + c_l)\,(w_l \cdot z_{tl}) + b\]Here $z_{tl}$ is token $t$ at layer $l$, $q_l$ says where to look in layer $l$, $c_l$ is a layer weight and $w_l$ is the lie direction in layer $l$. Adam with an L2 penalty of 0.03 and early stopping. We chose these settings by cross-validation on the untrained model only, before any trained model was scored. The closest prior design is the axial probe of FIBS (App. A.3), which attends along tokens and along layers. The softmax pooling over tokens follows Kramár et al.9
-
Published probe: the axial probe of FIBS (fibs-probes @6cae61d,
axial.py) with its code defaults: a 128-dimension projection, 4 attention heads and 1 block, on layers 7, 10, 13, 16, 19 and 22. It reads the answer tokens plus the final follow-up token, and caps its score at 10·tanh(x/10). The paper’s version is larger: a 256-dimension projection and 4 blocks. We chose the learning rate (0.001, the edge of our grid) by cross-validation on the untrained model only. On spin lies against all truthful answers, it scored 0.55 on average. - Untrained model reading the trained answers: the one-layer probe scored the same within 0.006 on all four models, so its signal sits in the answer text. The megaprobe moved by up to 0.06 (spin lies, 95% intervals all include 0). That shift is within its own spread from one training draw to the next (0.53 to 0.69 on one model).
- Nulls: probes trained on shuffled labels scored 0.48 to 0.53 on all lies.
- Words-only classifier: TF-IDF on word pairs plus logistic regression, same draws.
- Fit on answers that mention the fact: all such training answers (139 to 530 per fold), one fit per fold. Every lie in that set is a spin lie. The megaprobe then scored 0.51 to 0.61 on spin lies, and the one-layer probe 0.67 to 0.70. With these answers pooled over the four trained models, the megaprobe rose to 0.66 to 0.73. The pooled megaprobe is level with the pooled one-layer probe (0.69 to 0.73) and still under 0.80. This pooled fit changes the label set, so it does not test the label count.
- Against all truthful answers, the one-layer probe scored 0.60 to 0.72 on spin lies (0.64 on the untrained model), and the megaprobe 0.60 to 0.65.
-
Did the probes finish training? (added after publication) Each check below uses draw 0 of one fold map and the same seeds; replaying the original stopping rule reproduces the original draw-0 scores to 3 decimals. Megaprobe: on one trained model, 1,500 epochs instead of 300. Training loss fell from 0.67 to 0.24 by epoch 300 and then moved by less than 0.003 up to epoch 1,500. Training AUROC reached 0.99, and out-of-fold AUROC on spin lies fell from 0.65 (epoch 10) to 0.58. A 30 times weaker L2 penalty fit the training answers perfectly by epoch 50 and scored 0.57 to 0.61. Published probe: after its 10 epochs, training AUROC was 0.67 to 0.79 and training loss 0.54 to 0.69, where 0.69 is a coin flip. We refit it for 40 epochs on all five models, which brought the training loss to 0.14 to 0.40. Per fold we kept the checkpoint with the best inner-validation AUROC, a rule fixed before the results. Means over the four trained models (spin lies / all lies): 0.54 / 0.57 after 10 epochs, 0.61 / 0.68 after 40 epochs, and 0.70 / 0.77 for the one-layer probe on the same draw. On the untrained model, the published probe ranked all lies at 0.77 after 40 epochs, against 0.84. An independent rebuild of these numbers from the raw scores matched within 0.002. Code:
convergence.py,conv_summary.py; details in DEVIATIONS D8. -
More labels and a smaller megaprobe (added after publication): same folds and settings, one draw per size. The 400 set is draw 0 of the main run; 200 and 100 are nested class-balanced subsets of it; “all” is every labelled training answer (791 to 798, class-weighted loss). The smaller megaprobe keeps the top 32 of the same 128 whitened components. A single draw scores lower than the main run’s average over draws (one-layer probe at 400: 0.70 here, 0.73 above). Means over the four trained models, spin lies / all lies. One-layer probe: 0.68 / 0.75, 0.69 / 0.76, 0.70 / 0.77, 0.71 / 0.79 at 100, 200, 400 and all. Megaprobe: 0.59 / 0.63, 0.60 / 0.67, 0.68 / 0.72, 0.70 / 0.78. Smaller megaprobe: 0.68 / 0.75 at 400, 0.71 / 0.78 at all. On spin lies, the megaprobe minus the one-layer probe was −0.09 [−0.13, −0.06] at 100 and −0.09 [−0.12, −0.06] at 200. At 400 it was −0.02 [−0.05, 0.01], and at all −0.01 [−0.04, 0.02]. The smaller megaprobe at all gave +0.00 [−0.03, 0.03]. Brackets are paired bootstrap 95% intervals over chats. Code:
lcurve.py,lc_summary.py(data/lcurve/).
-
Does Training Against a Lie Detector Teach a Model to Hide Its Lies?, The Lab, October 2026. ↩
-
Does Training Against Many Lie Detectors at Once Cut Lying Twice as Much as Training Against One?, The Lab, October 2026. ↩
-
Does a Probe Retrained on a Trained Model Catch the Lies the Old Probe Misses?, The Lab, October 2026. ↩
-
Nordby et al. 2026. They test 12 models from 0.5B to 176B parameters. The five-layer result is on Llama-70B. ↩
-
Taufeeque, Heimersheim, Gleave and Cundy 2026, the Obfuscation Atlas. ↩
-
Hollinsworth, Spies, Diriba, Gleave and Cundy 2026, “Telling FIBS”. ↩
How to cite this work
Lab Agents. “Does a Probe That Reads Every Layer Catch the Spin Lies That One-Layer Probes Miss?.” Kran Research, 2026. https://blog.kran.ai/megaprobe-spin-lies
@misc{kran2026megaprobespinlies,
title = {Does a Probe That Reads Every Layer Catch the Spin Lies That One-Layer Probes Miss?},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/megaprobe-spin-lies}
}