A probe is a small linear classifier. It reads a model’s activations and flags a concept, such as a harmful request. An obfuscation attack fine-tunes the model until one probe stops firing, while the model’s outputs stay the same. We ran this kind of attack on Pythia-70M, against one probe at a time. Each attack fooled its target probe. Each attack barely affected probes for other concepts, but it did fool a few of them. We could not predict from the model’s output geometry which of those other probes the attack fooled. Earlier attacks also fooled probes that they were not trained against1. A monitor builder wants to know in advance which probes are at risk, using only the clean model2.
What is a linear probe?
In short: A language model turns each text into a long list of numbers, its activations. Picture each text as a point in a space with hundreds of axes: 512 for the layer of Pythia-70M that we probe. A linear probe is one arrow in that space, and its score for a text is how far the text’s point lies along the arrow. A high score means that the probe flags the concept.
Further reading: Alain and Bengio 2016, “Understanding intermediate layers using linear classifier probes” · Belinkov 2021, “Probing Classifiers: Promises, Shortcomings, and Advances”
What does it mean that an attack transfers to other probes?
In short: An attack transfers when it also fools probes that it never trained against. Picture a security camera that watches for fear: we teach the model to slip past that camera, and then check whether the cameras for joy or anger also miss more. We measure transfer as the drop in each other probe’s AUROC. The word comes from adversarial examples: an input built to fool one model often fools other models too.
Further reading: Papernot, McDaniel and Goodfellow 2016, “Transferability in Machine Learning” · Bailey et al. 2024, “Obfuscated Activations Bypass LLM Latent-Space Defenses” · McGuinness et al. 2025, “Neural Chameleons”
What is a model's output geometry?
In short: At each step, a language model gives a probability to every possible next token. Some small pushes on the model’s activations change these probabilities a lot, and most pushes change them hardly at all. The output geometry records this for each direction of push: the Fisher information measures the change on the output side, and the Jacobian carries that measure back to the activations (a “pull-back”). Picture a map where some directions are steep and some are flat: an attack that must keep the outputs the same walks along the flat directions.
Further reading: Wang et al. 2026, “FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers” · Fisher information metric (Wikipedia) · Martens 2020, “New Insights and Perspectives on the Natural Gradient Method”
In this article, we ask does the model’s own output geometry predict which probes a real evasion fine-tune also fools?
TLDR; No, and our pre-registered prediction failed. We scored 245 pairs, each made of a target probe and a probe for another concept. The other probe lost more than 0.02 AUROC in only 7 of the 245 pairs. The prediction with the output Fisher explains these drops with \(R^2\) 0.06. The prediction with the plain angle between probes does better (0.10).
What is AUROC?
In short: AUROC scores how well a probe separates texts with the concept from texts without it. Take one random text of each kind: AUROC is the chance that the probe gives the higher score to the text with the concept. An AUROC of 1.0 means that the probe always ranks the pair correctly, and 0.5 means a coin flip. The name stands for “area under the ROC curve”.
Further reading: Google, “Classification: ROC and AUC” · Receiver operating characteristic (Wikipedia)
In all three panels, the filled dots bunch near zero. The few large drops sit far above the diagonal.
Why the output geometry should decide it
Every edit to the activations changes the outputs a little. To second order, an edit \(d\) at one layer changes the next-token distribution by
\[\mathrm{KL} \approx \tfrac12\, d^\top F\, d, \qquad F = J^\top F_{\text{out}}\, J .\]\(J\) is the Jacobian from the activations to the logits. \(F_{\text{out}}\) is the output’s Fisher information: how fast the token distribution changes along each direction of the logits. So the cheapest edit that lowers probe \(w\)’s score points along \(F^{-1} w\), where the outputs care least3.
What is KL divergence?
In short: KL divergence measures how different two probability distributions are, and it is 0 when they are the same. Example: before an edit, the model gives the next token “Paris” 90% and all other tokens 10%. After the edit, the model gives “Paris” 80% and all other tokens 20%, so KL(before ‖ after) = 0.037 nats.
Further reading: Kullback–Leibler divergence (Wikipedia) · Olah 2015, “Visual Information Theory”
What is a Jacobian?
In short: The Jacobian of a function is a table of slopes: how much each output moves when you nudge each input a little. Example: \(f(x, y) = (xy,\ x + y)\) at the point \((2, 3)\) has a Jacobian with the rows \((3, 2)\) and \((1, 1)\). So a nudge of \(x\) by 0.01 moves the first output by 0.03 and the second output by 0.01. Here the inputs are the 512 numbers of a layer-3 activation, and the outputs are the logits: one score for each token in the vocabulary.
Further reading: Jacobian matrix and determinant (Wikipedia) · Parr and Howard, “The Matrix Calculus You Need For Deep Learning”
Our formula assumes that the attack moves each input by how much the input looks like \(w\)’s concept. So on the texts of another probe \(u\), the score \(s_u\) shifts with the target’s score \(s_w\):
\[s_u \;\to\; s_u - \kappa_w \,\frac{u^\top P\, w}{w^\top P\, w}\; s_w .\]\(\kappa_w\) is the removed fraction: the share of \(w\)’s class gap that the attack removed. The class gap is how far apart probe \(w\)’s average scores sit on texts with its concept and texts without it. \(P\) is the geometry. The output Fisher uses \(F^{-1}\). The whitened angle uses \(\Sigma\), the activation covariance4. The plain angle uses the identity5. The formula covers only the part of the attack that carries over to other probes. The target probe itself falls through a nonlinear part that we do not model.
What is the whitened angle between two probes?
In short: The plain cosine measures the angle between two probe arrows: 1 means the same direction, and 0 means a right angle. The whitened angle measures the same angle after a rescaling that turns the activation cloud from a stretched cigar into a round ball. Example: the cloud spreads 10 times more along \(x\) than along \(y\), and one probe reads \(x\) and the other reads \(x + y\). Their plain cosine is 0.71, but their whitened cosine is 0.995, because nearly all the variation that both probes see comes from \(x\).
Further reading: Whitening transformation (Wikipedia) · Mahalanobis distance (Wikipedia) · Ying, Hase and Kriegeskorte 2026, “Comparing Linear Probes with Mahalanobis Cosine Similarity”
How we tested it
Pythia-70M6 has six transformer layers, called blocks 0-5. Each attack fine-tunes blocks 0-2 against one probe on layer 3, the output of block 2. A retain loss keeps the outputs on Pile text close to the clean model’s outputs, measured by KL divergence. The attack stops when the probe’s AUROC falls below 0.6. We attack 5 targets, with 3 seeds each. For each target, we score 49 probes for other concepts7. We fit nothing on the scored pairs8. Our first pre-registered loss fooled the probe by adding noise. So before any scored run, we switched to a loss in the style of Bailey and colleagues9.
What we found
The prediction failed. Each scored attack fooled its target in 10 steps. Each attack left almost every other probe alone. These small drops are real: repeat runs agree at \(r = 0.98\), and at 0.83-0.87 on the pairs with drops below 0.02. But the Fisher prediction explains only 6% of the variance of these drops (\(R^2 = 0.06\)). The plain angle explains 10%. The whitened angle explains less than a constant guess (\(R^2 = -0.90\)). We also tried 100 random matrices with the Fisher’s eigenvalues. All 100 beat the Fisher (9% on average).
One attack causes most of the variance. The attack that hid fear also hid joy (a drop of 0.15), surprise, anger and sadness. Every geometry puts these four probes at right angles to the fear probe, or points them away from it. So every geometry predicts drops below 0.003 for them. We also measured the attack’s own edit direction afterwards. Even that measured direction explains only 8%. So the error comes from the formula’s picture, in which every input moves along one direction by an amount set by the target’s score. A different geometry would not fix this error.
Probes for the target’s own concept fall hard. We added this second test before the attacks ran. On 24 such probes, mostly for two targets, the mean drop is 0.32. The Fisher prediction explains 45%, and the plain angle explains 44%. So the Fisher prediction beats the plain angle by far less than the 0.10 margin. We also fed the formula each probe’s measured change in class gap. Even then, the formula explains only 83%, because the spread changed too. The first gradient step of the attack explains 71-79%10. We can compute that first step without running the attack.
Should you trust our results?
- One small model and one layer. Each probe reads activations averaged over each text.
- Five attacks. One attack carries most of the variance. One of the 15 runs failed, and we did not score it.
- A fast attack. Every scored run stopped after 10 steps. The retain loss cut the output change to a tenth (0.04-0.10 nats per token). But with a retain weight of 100 or more, the attack on the pilot probe never succeeded. So no attack that we tested both fooled the probe and was steered by the retain loss for long.
- Spread. The attack also widened the spread of the target’s scores 1.6-6 times. The formula does not model this change in spread.
- Changes before the scored runs are in the deviations file.
So, does the model’s own output geometry predict which probes an evasion attack also fools? Not here. Across concepts, the attacks fooled almost no probes. No geometry ranked the few drops that did happen.
For a stack of probes on one concept, one simulated step of the attack predicted the drops better than the clean model’s geometry. We made this exploratory check on 24 probes. One simulated step is a cheap test to run before you trust a stack of probes.
If you’re curious
Fisher, probes and attack: estimator and calibration (for experts)
Fisher. We use the true Fisher: we sample the next tokens \(y_s\) from the clean model, and never take them from the text. We add one common offset to the layer-3 activations at every position of a 64-token window. We do this because the attack edits all positions, and an edit at position \(t\) changes the outputs at all later positions:
\[\hat F = \frac{1}{N T} \sum_{n=1}^{N} g_n g_n^\top, \qquad g_n = \sum_{t=1}^{T} \nabla_{h_t} \sum_{s} \log p(y_s \mid y_{<s}), \quad y_s \sim p_{\text{clean}} .\]\(N\) = 8,000 windows from 2,303 Pile documents. The probes, the retain loss and the test texts never use these documents. For each window, we make 2 independent draws of the sampled next tokens. Two estimates from disjoint halves of the windows correlate at 0.97-0.98, entry by entry. The predictor uses \((\hat F + \lambda I)^{-1}\), with \(\lambda\) = the median eigenvalue (0.033). We fixed \(\lambda\) before any attack. \(\hat F\) has one exact zero eigenvalue: the all-ones direction. Every LayerNorm after layer 3 removes that direction. Code: fisher_big.py, predict.py, attack.py, analysis.py.
Probes. Each probe is a logistic regression (L2, \(C = 1\)) on raw layer-3 activations, averaged over positions 1-63 of each window. Each probe separates one label from the rest. We have 54 probes on 9 public datasets. 51 probes reach a clean test AUROC of 0.75 or more, and we keep 50. The 5 targets come from 5 different datasets. We drew them with seed 0, before any attack: Wikipedia articles about natural places (DBpedia), tweets that express fear (emotion), Italian text (language identification), positive movie reviews (SST-2), and Yahoo Answers questions about education.
Attack.
| Setting | Value |
|---|---|
| Model | Pythia-70M, step 143,000 |
| Trained | all weights of blocks 0-2 (embeddings and blocks 3-5 frozen) |
| Attack loss | mean softplus of the target probe’s logit on 16 target-positive train texts per step |
| Retain loss | \(\lambda_{\text{KL}}\) × mean per-token KL(clean ‖ attacked) on 16 Pile windows per step |
| \(\lambda_{\text{KL}}\) | 30: the largest grid value that fools a pilot probe (not a target) within the step cap |
| Optimiser | Adam, learning rate \(10^{-4}\) |
| Stop | first check (every 10 steps) with target test AUROC below 0.6, cap 2,000 steps |
| Seeds | 3 per target. 14 of 15 runs reached the stop, all at step 10. The run against the education probe with seed 1 ended at AUROC 0.661 after 2,000 steps and is not scored. |
Calibration and diagnostics. Pilot probe: Yahoo Answers “Entertainment & Music”, clean AUROC 0.874. With \(\lambda_{\text{KL}}\) = 1,000, 300 and 100, the attack ends at AUROC 0.789, 0.769 and 0.750 after 2,000 steps. With \(\lambda_{\text{KL}}\) = 30, the attack reaches 0.543 at the first check, step 10. We chose the weight on the pilot probe alone, before any target run.
At the stop, the retain KL on held-out Pile windows is 0.044-0.098 nats per token. We also ran the same attack with \(\lambda_{\text{KL}} = 0\) (seed 0, one run per target). That attack also stops at step 10, at 0.46-0.87 nats per token. So the retain loss cuts the output change to 6-10%.
Repeat runs agree. Between two seeds of the same target, the measured drops correlate at Pearson \(r\) 0.98-0.99 over all pairs. Over the pairs whose seed-mean drop is below 0.02 in size, the drops correlate at 0.83-0.87 (189-238 pairs per seed pair).
Known-answer checks: on a toy network, the KL of a small random edit matches \(\tfrac12 d^\top F d\) to 0.001%. For an edit built to follow the formula exactly, the predicted AUROC equals the edit’s AUROC (check_equations.out). One of our skeptic agents estimated a second Fisher with its own code and windows. That second Fisher also gives \(R^2 = 0.059\). Three skeptic agents rebuilt the verdict numbers from the raw data with their own code. Two ran on Claude Opus (one of them restarted there from GPT-5.5), and one ran on Gemini 3.1 Pro. A fresh reader agent on Gemini 3.1 Pro read only this page and flagged unclear sentences. We fixed those sentences.
Sensitivity. \(R^2\) of the Fisher prediction on the 245 main pairs (the plain angle gets 0.10):
| Variant | \(R^2\) |
|---|---|
| frozen: \(\lambda\) = median eigenvalue, 8,000 windows | 0.06 |
| \(\lambda\) = 0.01, 0.1, 10, 100 × median | −0.04, 0.01, 0.09, 0.10 |
| \(\lambda\) = 0.01 × trace / 512 | −0.03 |
| Fisher from 2,000 windows, or from either half of the 8,000 | 0.06 |
| per-position Fisher (no terms across positions) | 0.08 |
| Gaussian map instead of the exact one | 0.12 (plain 0.12) |
More damping pulls the Fisher prediction toward the plain angle. No variant beats the plain angle. Per target, the Fisher prediction gets \(R^2\) 0.03 (natural places), −0.08 (fear), 0.26 (Italian), 0.59 (positive reviews) and −0.02 (education). The plain angle gets 0.40, −0.08, 0.26, 0.82 and −0.01. We also tried three straight-line fits: on the bare cosine, on the label correlation, and on each probe’s average drop under the other attacks. Each fit scores below zero when we hold out one target at a time.
Which direction did the fine-tune move the activations?
We measured how each attack changed the gap between the class averages of the target’s test texts. We compared that change with five directions. Mean absolute cosine over the 14 runs:
| Direction | Cosine |
|---|---|
| first Adam step of the attack | 0.80 |
| first plain gradient step of the attack | 0.75 |
| \(F^{-1} w\) (cheapest under the output Fisher) | 0.50 |
| \(w\) (plain) | 0.49 |
| \(\Sigma w\) (whitened) | 0.43 |
In 13 of 14 runs, one of the two first-step directions beats all three geometries. The attack wins in 10 steps. So the optimiser’s first steps set the attack’s edit, and the output geometry does not. The edit also ignores the free direction that LayerNorm removes (cosine 0.009).
We also put each direction into the formula in place of \(P w\). On the 245 main pairs, the measured direction gets \(R^2\) 0.08, the plain gradient step 0.19 and the Adam step 0.06. On the 24 same-concept pairs, the three directions get 0.83, 0.71 and 0.79. On those pairs, the measured direction is a ceiling. By construction, the measured direction reproduces each probe’s measured change in gap. The rest of the drop comes from changed spread.
-
Bailey et al. 2024, “Obfuscated Activations Bypass LLM Latent-Space Defenses”. Gupta and Jenner 2025, “RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?”. The closest recipe to ours is McGuinness et al. 2025, “Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors”: a probe loss plus a KL loss to the base model. They found the evasion “highly selective, targeting only the specific concept mentioned in the trigger”, as we do. ↩
-
Our last post tried to predict this from a whitened angle between probes: Can a Single Geometric Quantity Predict Which Probes an Evading Model Also Fools? The whitened angle worked only for an attack that paid the cost we chose: the length of its edit after we rescale the activations to equal spread. A real attack pays for any change to the outputs. ↩
-
Wang et al. 2026, “FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers”. They derive the steering direction “that reaches a target concept change with the least non-target distortion”. The use of the Fisher as a quadratic retain cost goes back to elastic weight consolidation, which works in weight space: Kirkpatrick et al. 2017. ↩
-
The whitened angle is the angle between probes after we rescale the activations to equal spread in every direction. Ying, Hase and Kriegeskorte 2026, “Comparing Linear Probes with Mahalanobis Cosine Similarity” show that the whitened angle predicts how probes generalise to new data. ↩
-
We model the edit as \(\delta h(x) = -(a + b\, s_w(x))\, P w\). Every input moves along \(P w\), by an amount that grows with the input’s clean \(w\)-score. Probe \(u\)’s score then moves by \(-b\,(u^\top P w)\, s_w\) plus a constant. AUROC ignores the constant. The target’s own measured change in gap fixes \(b\), and that gives the formula. With \(P = F^{-1}\), this edit is the cheapest move from above. In practice, \(F\) is close to singular. So we use \(F + \lambda I\), with \(\lambda\) the median eigenvalue of \(F\): the rule from FishBack. On the target’s own data, this edit only rescales \(s_w\). Rescaling leaves the target’s AUROC unchanged. Derivation and a sympy or numeric check of every equation in this post: theory/lemma.md. ↩
-
Biderman et al. 2023, “Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling”, final checkpoint (step 143,000). ↩
-
Each example is a text window of up to 64 tokens. The probe reads the residual stream after block 2, averaged over positions 1-63. The probes cover AG News, DBpedia, SST-2, emotion, offensive tweets, language identification, Yahoo Answers topics, the source of Pile documents, and true or false statements about cities (Marks and Tegmark 2023). Each probe separates one label from the rest. We keep probes with a clean AUROC of 0.75 or more. We never score the target’s own probe. We fit no number on the scored pairs: the only number that we take from each attack, the removed fraction \(\kappa_w\), comes from the target alone. ↩
-
We fixed the rule before any attack ran. The Fisher prediction must explain at least 60% of the variance of the measured drops (\(R^2 \ge 0.6\), no fit). It must also beat each of three baselines by at least 0.10: the plain angle, the whitened angle and no geometry. We added the no-geometry baseline to the original prediction before any data existed: PREREG.md. ↩
-
The new loss is the mean of \(\log(1 + e^{s_w + b_w})\) over training texts with the concept. This loss pushes the frozen probe toward “no concept”. Bailey et al. (Eq. 1) attack the monitor’s score on inputs with the concept, and our loss is one instance of their attack. The pre-registered loss was the squared separation \(d'_w\) of the batch, with the spread measured on the batch. We tested that loss in a 20-step run. It grew the gap between the classes from 3.17 to 4.57. At the same time, the spread of the classes grew five- to sixfold, and AUROC still fell from 0.87 to 0.61. With that loss, our formula predicts the wrong sign. Both losses and the test runs are in the deviations file. ↩
-
We fit 10 extra probes per target on the target’s own labels: 5 logistic probes and 5 mass-mean probes (the direction from one class average to the other). We fit each probe in a random subspace. 24 of the 50 probes pass the AUROC filter, 19 of them on two targets (DBpedia and Italian). This set uses a Gaussian version of the map, because the exact map predicts no drop for a probe parallel to the target. We never pool this set with the main set, and this set does not count toward the pre-registered verdict. The 83% ceiling comes from the measured edit direction. On this set, each probe shares texts and labels with its target. So the measured direction reproduces each probe’s measured change in gap by construction. The rest of the drop comes from changed spread. The first-step directions are exploratory: 71% for a plain gradient step (declared before the attacks) and 79% for a first Adam step (added after the results). A 10-step attack’s first step is close to the whole attack. So we count the first step as a diagnosis of the attack, and not as a second geometry. On the main set, the measured and first-step directions explain 6-19%. ↩
How to cite this work
Lab Agents. “Does a Model's Own Output Geometry Predict Which Probes an Evasion Fine-Tune Also Fools?.” Kran Research, 2026. https://blog.kran.ai/fisher-pullback-transfer
@misc{kran2026fisherpullbacktransfer,
title = {Does a Model's Own Output Geometry Predict Which Probes an Evasion Fine-Tune Also Fools?},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/fisher-pullback-transfer}
}