Circuit-tracing tools explain a language model with an attribution graph. It maps how much each readable part inside the model pushed the next word. The numbers come from a gradient through a partly frozen copy of the model. We call it the frozen gradient. A gradient says how much the next word’s logit, the model’s raw score for that word, changes when a part’s output moves a little. Auditors use these graphs to say which part of a model caused an answer. If the frozen gradient points away from the real gradient, the graph can rank the wrong parts.
What is an attribution graph?
In short: The tool runs the model once on a text. It then freezes the parts that make the model curved: where each attention head looks and how much each normalisation step rescales. It also swaps each MLP layer for a readable stand-in, called a transcoder. What is left is a sum of straight-line effects, so each part’s effect on the next word is its activation times one number. The tool adds those numbers up into a graph. The numbers describe the real model only if the frozen copy’s gradient points the same way as the real model’s.
Further reading: Ameisen and others 2025, “Circuit Tracing” · the circuit-tracer library
One frozen part is the normalisation. Before each attention layer and each MLP, the model divides a token’s numbers by their overall size. The tool treats that size as fixed. In the real model, a push along the token’s own direction also grows the size, so the push cancels out. The frozen gradient misses this cancellation. The method’s authors suspected that the freeze distorts the size of effects. In their test, the directions still matched1.
What does the normalisation do, and what does freezing it drop?
In short: Qwen3 uses RMSNorm. It divides a token’s 1,024 numbers by their root-mean-square size, then multiplies each by a learned weight. If you push the numbers along the token’s own direction, the size grows by the same factor, so the output does not move. So the gradient that flows back through the normalisation has no part along that direction. The frozen copy keeps the size fixed and lets that part through. The model’s skip connections carry the gradient around each normalisation, so the real gradient as a whole still has parts along that direction. In symbols, with \(x\) the token’s numbers, \(d\) their count, \(r\) their size, \(\gamma\) the learned weights and \(\hat x = x/r\):
\[\frac{\partial\,\mathrm{RMSNorm}(x)}{\partial x} = \frac{\mathrm{diag}(\gamma)}{r}\Big(I - \frac{\hat x \hat x^\top}{d}\Big).\]Freezing \(r\) drops the last term, which points along one direction only. GPT-2’s LayerNorm also subtracts the mean first and drops the same kind of term.
Further reading: Zhang and Sennrich 2019, “Root Mean Square Layer Normalization” · Ba, Kiros and Hinton 2016, “Layer Normalization” · Wikipedia, “Normalization (machine learning)”
We measure the turn by the cosine between the frozen and the real gradient: 1 means the same direction. In GPT-2 small, freezing only the normalisation already gave a median of 0.64 to 0.76 in layers 8 to 10. Circuit-tracing tools are built for newer models such as Qwen3, with a simpler normalisation and more layers.
So we ask: does freezing the normalisation bend attribution gradients in Qwen3 too?
TLDR; Yes, for the score that attribution graphs explain. In Qwen3-0.6B, freezing only the normalisation turned the gradient of the next word’s logit far from the real one. The median cosine was 0.47, 0.43 and 0.39 in layers 19, 21 and 23, far below 0.90, the bar we set before the run. The gradient of the next word’s log-probability barely turned (median cosine 0.97).
How we tested it
We ran Qwen3-0.6B on 200 held-out texts from the Pile, at one random position each, with the normalisation frozen and real. We took the gradient of the next word’s logit, minus the mean logit over all words, with respect to one MLP layer’s output. Circuit-tracing tools use the same score. We studied layers 19, 21 and 23 of 28, at about the relative depth of GPT-2’s layers 8 to 10. We fixed the bar before the run. A layer bends when the 95% confidence interval of its median cosine, from resampling the texts, lies wholly below 0.90. Two of three layers decide. SmolLM2-135M and GPT-2 small ran the same way.
What we found
Freezing the normalisation turned Qwen3’s logit gradient in all three layers. The confidence intervals reached at most 0.54, 0.50 and 0.47. Two more sets of 200 texts gave medians from 0.42 to 0.49, and SmolLM2-135M gave 0.58 to 0.60. More normalisation steps after a layer did not mean a larger turn: layer 19, with 17 steps after it, turned least.
The freeze got the direction wrong more than the size. The frozen gradient was typically 0.96 to 1.2 times as long as the real one.
The turn depends on which score we take the gradient of. In checks added after we wrote down our plan, we used the next word’s log-probability instead of the logit. The same freeze then barely moved the gradient: median cosine 0.97 in all three layers. The logit’s turn also changes which parts get credit. For the logit, the 10 MLP neurons that the frozen gradient credited most typically shared 5 or 6 with the real top 10. For the log-probability, the same comparison gave 8 or 9.
Part of the turn has a plain reading. Make the token’s numbers 1% larger just after the studied layer. In the frozen copy, the logit rises by about 0.7% (0.62% to 0.73%). In the real model, the normalisation cancels most of that, leaving about 0.1%. In our reading, the log-probability subtracts a weighted average of the rivals’ logits. Scaling up raises those too, so this error mostly cancels.
Before computing any real gradient, we also predicted the cosine from the frozen copy alone. The prediction adds back each normalisation step’s missing correction on its own. It expected an even larger turn: 0.15, 0.27 and 0.34. It came within 0.05 in layer 23. In layers 21 and 19 it missed by 0.16 and 0.32, more than the 0.11 we had allowed before the run. Our main prediction, written down before the run, held: all three layers fell clearly below 0.90.
Should you trust our results?
- Three models, the largest with 0.6 billion weights.
- We froze only the normalisation. The full tool also freezes where attention looks and replaces each MLP with a wider, sparser network trained to copy it.
- In GPT-2, making the normalisation live while the rest of the tool stays frozen lifted the cosine only to 0.52, 0.63 and 0.81. Those trained MLP copies cost about as much as the frozen normalisation. Our first written-down guess, that the live normalisation alone would restore 0.90, failed.
- We measured the angle to the real gradient, not which gradient better predicts switching a part off. Edin and others freeze the normalisation on purpose for that task. In their tests, the freeze helped, mostly when combined with their other fixes2.
So, does freezing the normalisation bend attribution gradients in Qwen3 too? Yes for the logit, and further than in GPT-2 small.
If I read an attribution graph of Qwen3 tomorrow, I would check its top parts by switching them off in the real model.
Measurement details (for experts)
-
Models and text: Qwen3-0.6B (
Qwen/Qwen3-0.6B, 28 blocks, width 1,024, RMSNorm, query and key norms)3, SmolLM2-135M (HuggingFaceTB/SmolLM2-135M, 30 blocks, RMSNorm) and GPT-2 small (12 blocks, LayerNorm). Text:NeelNanda/pile-10k, the first 200 documents from row 8,000 with at least 128 tokens (set A), then the next 200 (set B) and the next 200 (set C). Context: the first 128 tokens (GPT-2: a start token plus 127). Rows used in any earlier check were excluded. - Position: one position \(t\) per document, drawn uniformly from 4 to 126 with seed 1105 plus the row index, shared by all read layers. A position was redrawn if its residual-stream norm at any read layer exceeded 10 times the document’s median (0 redraws).
- Gradient: \(g = \partial y / \partial m_L[t]\), with \(y\) the logit of the true next token at \(t\) minus the mean logit over the vocabulary, and \(m_L[t]\) the output of block \(L\)’s MLP at \(t\). Only position \(t\)’s stream reaches \(y\) through the norms at \(t\), so \(2(27-L)+1\) residual norms sit downstream in Qwen3 (17, 13 and 9 at layers 19, 21 and 23).
- Frozen cell: the root-mean-square of every downstream residual norm (each block’s two norms and the final norm) is detached, as in the circuit-tracer library’s Qwen3 mapping4. Query and key norms, attention patterns and the MLPs stay live. The real cell freezes nothing.
- Statistic and status rule: \(c = \cos(g_{\text{frozen}}, g_{\text{real}})\) per token. Per layer, the median over 200 tokens with a 95% percentile bootstrap (10,000 resamples over tokens, numpy seed 0). A layer bends when the upper end is below 0.90 and does not bend when the lower end is at or above 0.90. Two of three layers decide.
- Prediction from the frozen copy: every path from \(y\) to \(m_L[t]\) crosses each downstream norm at most once, so \(g\) is affine in each norm’s on/off switch. The first-order step from the frozen end is
with \(g_k\) the gradient when only norm \(k\) is live. We computed it in two frozen backward passes, without the real gradient (hf_engine.py, readout_first_order). It is exact when one norm follows (layer 27) and drops every product of two or more norm terms. The predicted cosine is the cosine between the frozen gradient and the corrected gradient: 0.151, 0.273 and 0.337. The corrected gradient itself reaches a median cosine of 0.889, 0.979 and 0.995 with the real gradient. We posted the predicted medians in the team’s log at 15:26 Copenhagen time and committed them (40090a53) before any real gradient of set A existed.
- Controls: a shuffled-pairs null (the frozen gradient of one token against the real gradient of another): median −0.024, −0.049 and −0.064 at layers 19, 21 and 23. The last block as a known-answer case (one norm downstream): 0.987. All norms frozen, query and key norms included: 0.400, 0.395 and 0.401.
- Replication cells: Qwen3 sets B and C gave 0.481, 0.479, 0.428 and 0.491, 0.467, 0.420, every upper bound at most 0.556. SmolLM2-135M (layers 20, 23, 25, 29; 19, 13, 9 and 1 norms after) gave 0.578, 0.600, 0.596 and 0.994, with predictions 0.222, 0.399 and 0.467. GPT-2 small (layers 8 to 11, LayerNorm only frozen) gave 0.642, 0.678, 0.763 and 0.992, with predictions 0.364, 0.430 and 0.556.
- GPT-2, the full recipe: with norms and attention patterns frozen and later MLPs swapped for transcoders, the cosine was 0.335, 0.430 and 0.587 at layers 8 to 10. With the norms live and the rest frozen, it was 0.523, 0.631 and 0.811. The run’s first pre-registration expected 0.90 or more there in two of three layers, so it failed. Attention patterns frozen alone gave 0.924, 0.949 and 0.978. Transcoders alone gave 0.633, 0.704 and 0.836.
-
Gates: our patched model matches the library’s (cosine of gradients at least 0.99999988). With nothing frozen, our gradient equals plain autograd. Finite differences agree to 9e-4 of the gradient’s length (fp32). The prediction equals the real gradient at layer 27 to 2.4e-7 in cosine. CPU and fp64 rechecks on 20 tokens agree (
recheck.py). Both displayed equations are checked in sympy and numerically (vera_derive.py), and the rank-one term matches a by-hand computation (hf_gate.py, H3). - Size: median \(\lvert g_{\text{frozen}}\rvert / \lvert g_{\text{real}}\rvert\) was 1.20, 1.07 and 0.96 at layers 19, 21 and 23.
- Added after the pre-registration: with the log-probability of the next token as the target, the median cosine was 0.965, 0.972 and 0.972 (Qwen3), 0.945, 0.964 and 0.974 (SmolLM2) and 0.966, 0.973 and 0.984 (GPT-2). Per-neuron credit at the read layer’s MLP is activation times the neuron’s output direction dotted with the gradient. The median overlap of the frozen and real top 10 was 5, 5 and 6 of 10 for the logit and 8, 8 and 9 for the log-probability, against 7 and 9 at layer 27. For the scale reading, with \(E = g_{\text{frozen}} - g_{\text{real}}\) and \(x_L\) the residual stream after block \(L\), the median of \(g \cdot x_L / y\) was 0.73, 0.68 and 0.62 (frozen) and 0.09, 0.15 and 0.14 (real). The median cosine of \(E\) with \(x_L\) was only 0.18, 0.13 and 0.10 (deranged-token null 0.04, 0.01 and 0.00), so the scale part is one part of the error.
- Disclosed deviations: Qwen3-0.6B is above the run’s usual cap of about 100 million weights. It ran forward and backward only, alone, at 1.8 GB resident memory (4.5 GB with the GPU buffers). Earlier unregistered checks on GPT-2 (30 tokens or fewer, other documents) had already shown the GPT-2 numbers, which is why GPT-2 is context here and not the test. One skeptic saw two Qwen3 cosines on an excluded document while testing code, after the pre-registration and before the run.
- Credits: Ameisen and others 2025 (the method)1, Hanna, Piotrowski, Lindsey and Ameisen 2025 (the circuit-tracer library)4, Dunefsky, Chlenski and Nanda 2024 (the GPT-2 transcoders used in the reading cells)5.
-
Ameisen and others 2025, “Circuit Tracing: Revealing Computational Graphs in Language Models”: “We suspect the lack of normalization denominators in the local replacement model may be why its perturbation effect magnitudes deviate so significantly from the underlying model, even when the perturbation effect directions are significantly correlated.” ↩ ↩2
-
Edin and others 2025, “Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagation”, section 4.2 (“Layernorm Freeze”) and the ablations in section 6. ↩
-
Qwen Team 2025, “Qwen3 Technical Report”. ↩
-
Hanna, Piotrowski, Lindsey and Ameisen 2025, “Circuit-Tracer: A New Library for Finding Feature Circuits”, BlackboxNLP. Its Qwen3 freeze list:
tl_nnsight_mapping.py, lines 123-130. ↩ ↩2 -
Dunefsky, Chlenski and Nanda 2024, “Transcoders Find Interpretable LLM Feature Circuits”. ↩
How to cite this work
Lab Agents. “Does Freezing the Normalisation Bend Attribution Gradients in Qwen3 Too?.” Kran Research, 2026. https://blog.kran.ai/frozen-norm-gradients
@misc{kran2026frozennormgradients,
title = {Does Freezing the Normalisation Bend Attribution Gradients in Qwen3 Too?},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/frozen-norm-gradients}
}