Circuit-tracing tools explain a language model with an attribution graph. The graph is a map of readable units inside the model and how much each one pushed the next word. An auditor reads claims like “switching this unit off lowers the logit of the word American by 1.9”. The logit is the model’s raw score for a next word. That claim is only useful if the real model agrees.
The units come from transcoders. An MLP layer has a few thousand neurons, and many of them do several unrelated jobs. A transcoder is a small network trained to give the same output as one MLP layer. Its many features each fire rarely and for one readable thing. The graph tools replace every MLP layer with its transcoder.
What is a transcoder?
In short: A transcoder takes the same input as one MLP layer and is trained to give the same output. It has a wide hidden layer (24,576 features for GPT-2 small, against 3,072 neurons), and a penalty keeps most features off on each token. Each feature reads its input along one direction (its encoder) and writes along one direction (its decoder). Because few features fire at once, each one tends to mean one thing, which makes a graph built from them readable.
Further reading: Dunefsky, Chlenski and Nanda 2024, “Transcoders Find Interpretable LLM Feature Circuits” · Ameisen and others 2025, “Circuit Tracing”
Shabalin and Piotrowski found that in 13 open transcoders, many late-layer features are near copies of one MLP neuron1. They read and write along that neuron’s directions. We call them neuron features. In six transcoders, the authors compared how each layer responds to a tiny change in its input with how the real MLP responds. Almost all of the match came from neuron features. They list real interventions as untested. We ask whether the mismatch reaches the model’s output.
So we ask: do transcoder graphs predict the logit only for features that copy a neuron?
TLDR; Not settled. In all four late layers of GPT-2 small, the transcoder matched the real layer better for neuron features than for features that copy no neuron. Agreement (R², where 1 is perfect) was 0.45 to 0.94 against −1.13 to 0.59. No layer cleared both bars we set in advance, and dropping 2 of 300 features moved one layer across a bar. In three of the four layers, halfway features did about as well as neuron features. Most big misses came from features that copy no neuron.
In no row do neuron features pass 0.7 while features that copy no neuron stay under 0.4.
How we tested it
We used the public transcoders for the last four layers of GPT-2 small2. For each feature, we took tokens where it fires on held-out text. We nudged the MLP layer’s input just enough to switch that feature off. We fed the nudged input to the transcoder and to the real MLP layer. Then we turned each output change into a change in the true next word’s logit, with the real model’s gradient. This scores only the swapped layer. The graph’s own gradient has its own section below. An R² below 0 means the transcoder is further off than always guessing the average change.
What is R²?
In short: R² says how much of the real changes the transcoder gets right. Take all nudges in a group. For each, \(s\) is the real layer’s change in the logit and \(\hat s\) is the transcoder’s. Then R² \(= 1 - \sum (s - \hat s)^2 / \sum (s - \bar s)^2\), where \(\bar s\) is the average real change. R² is 1 when every guess is exact, 0 when the guesses are no better than always saying \(\bar s\), and below 0 when they are worse than that.
Further reading: Coefficient of determination (Wikipedia)
In each layer, we drew three sets of 100 features from each of three groups. Cosine similarity measures how well two directions line up: 1 is the same direction, 0 is unrelated. Neuron features line up with one neuron at 0.9 or more, on input and output. Features that copy no neuron stay under 0.5 on both. Halfway features sit between the two. Before the runs, we fixed the bars for a clear yes. Neuron features needed 0.7 or more and features that copy no neuron 0.4 or less, in 3 of 4 layers.
What we found
Neuron features came out ahead of features that copy no neuron in every layer, but never by enough. Their R² was 0.66, 0.94, 0.45 and 0.63 in layers 8 to 11, against −1.13, 0.59, −0.15 and −0.55. If the group labels did not matter, a gap this steady in all four layers would turn up about 2 times in 1,000. We added this check after the runs. No single layer is clear on its own.
In layers 8, 9 and 11, halfway features did about as well as neuron features (0.66, 0.86 and 0.64 against 0.66, 0.94 and 0.63). We did not pre-register this comparison. Features that copy no neuron had the worst misses. In layer 11, 67 of their 2,246 nudges missed by more than 1 logit, against 3 of 2,400 for neuron features. A few such misses decide each R². Dropping the 2 worst of 300 neuron features lifted layer 8 from 0.66 to 0.71, past the bar.
Closeness to one neuron is only a weak filter for single features. We ranked 3,600 features by closeness and by their typical miss, after taking out firing rate, activation size and weight norms. The rank correlation was only −0.094. Our rule counted anything weaker than −0.1 as a fail, and this fell 0.006 short. Each layer alone gave −0.13 to −0.16, a weak trend in the predicted direction. In the figure’s bottom strip, the typical miss sits near 0.5 below cosine 0.4 and near 0.4 from cosine 0.5 up. Neuron features gain nothing extra.
Our prediction, fixed before the runs, failed: only layer 9 had neuron features at 0.7 or more. It did not fail in the other direction either: features that copy no neuron did worse in every layer. So the data say neither yes nor no.
The graph’s own gradient missed more
We did not pre-register this check. To score a change, an attribution graph uses the gradient of a frozen copy of the model. Where attention looks and how much each layer rescales its numbers stay fixed, and later MLP layers become transcoders. In layers 8 to 10, that gradient pointed away from the real one (median cosine 0.34 to 0.61). With it, our nudges scored R² between −0.7 and −146 against the real model’s change. In layer 11, only one rescaling step follows. The cosine was 0.99, and R² was 0.57 for neuron features and −2.2 for the other two groups.
What is an attribution graph's prediction?
In short: The tool runs the model once on a text. It then freezes the parts that make the model curved, such as where each attention head looks and how much each LayerNorm rescales. What is left is a sum of straight-line effects. So each feature’s effect on the next-word score is its activation times one number, and the tool can add those numbers up. A finite change, like switching a feature off, can still differ from these straight-line numbers.
Further reading: Ameisen and others 2025, “Circuit Tracing” · the circuit-tracer tool · an interactive graph on Neuronpedia
Should you trust our results?
- One small model with a plain MLP, one transcoder release, four layers.
- Three draws of 100 features per group, so more draws could tip a layer.
So, do transcoder graphs predict the logit only for features that copy a neuron? Not settled. Neuron features beat features that copy no neuron in every layer, but a handful of features decide the score.
If we drew a transcoder graph of GPT-2 small tomorrow, we would first check its gradient against the real model’s. Which features copy neurons would come second.
Measurement details (for experts)
-
Model, transcoders, text: GPT-2 small (
openai-community/gpt2) with thepchlenski/gpt2-transcodersrelease2 for layers 8 to 11 (24,576 ReLU features each). A transcoder reads \(x_n\), the output of the layer’s second LayerNorm before its gain and bias (blocks.L.ln2.hook_normalized), and writes the centred MLP output. Text:NeelNanda/pile-10k, documents 5,000 onward, 2,000 windows of one start token plus 127 tokens. No training. - Groups: for feature \(f\) with encoder \(e_f\) and decoder \(d_f\), the input cosine is the largest \(\lvert\cos(e_f, \gamma \odot W_{\text{in}}[:, n])\rvert\) over neurons \(n\) (LayerNorm gain \(\gamma\) folded in). The output cosine is \(\lvert\cos(d_f, W_{\text{out}}[n])\rvert\) for that same neuron, with \(W_{\text{out}}\) rows centred. Neuron features have both at 0.9 or more. Features that copy no neuron have both maxima under 0.5, over any neurons. Halfway features have the smaller cosine in [0.5, 0.9), 25 per 0.1 bin. Shabalin and Piotrowski use 0.7 on both sides. Pooling our neuron sample with the 0.7-0.9 halfway samples gives R² 0.76, 0.94, −0.12 and 0.68.
- Sampling: per layer, group and draw, 100 features active on the windows, up to 8 active tokens each. Three disjoint draws (seeds 0, 1, 2). 0 nudges dropped. About 2,400 nudges per group and layer (2,246 to 2,400).
- Nudge: \(\delta u = -a_f\, e_f / \lVert e_f \rVert^2\) added to \(x_n\) at layer \(L\) only, with \(a_f\) the feature’s activation. It sets \(f\)’s pre-activation \((x_n - b_{\text{dec}})\cdot e_f + b_{\text{enc}}\) to exactly 0 (checked on every nudge). The median nudge is 0.6 to 1.9% of \(\lVert x_n \rVert\).
- Scores: \(g\) is the exact gradient of the true next token’s centred logit (logit minus the mean logit over the vocabulary) with respect to the layer’s MLP output, in the real model. Transcoder change \(\hat s = g \cdot (\mathrm{TC}_L(x_n + \delta u) - \mathrm{TC}_L(x_n))\). Real-layer change \(s = g \cdot (\mathrm{MLP}_L(x_n + \delta u) - \mathrm{MLP}_L(x_n))\). R² about the line \(\hat s = s\) is \(1 - \sum (s - \hat s)^2 / \sum (s - \bar s)^2\), pooled over a cell’s nudges. The 95% interval bootstraps features (2,000 draws). A feature’s typical miss is the median over its tokens of \(\lvert s - \hat s \rvert / (\lvert s \rvert + \lvert \hat s \rvert)\).
- Read-out gate: \(s\) must predict the real end-to-end change in the centred logit (real MLP, real later layers) with R² of 0.7 or more, else the layer does not count. It gave 0.82, 0.84, 0.79 and 0.88. The forward pass with the nudge cached matched the full forward pass exactly.
- Status rule (frozen): a layer says yes if neuron features reach 0.7 or more and features that copy no neuron 0.4 or less. It says no if features that copy no neuron reach 0.7, or neuron features fall under 0.4, or the gap is under 0.1. A layer’s verdict counts only if at least 2 of 3 draws agree with it. Yes in 3 of 4 layers is CONFIRMED, no in 2 of 4 is REJECTED, else UNCONFIRMED. All four layers said neither.
- Gap against shuffled labels: a two-sided test shuffles group labels between neuron features and features that copy no neuron. It gave p 0.17, 0.048, 0.041 and 0.019 in layers 8 to 11, uncorrected for 4 layers (Holm: smallest 0.076). Fisher’s combination over the 4 layers gives p 0.002. The skeptic’s own shuffle gave 0.20, 0.056, 0.034 and 0.032.
- Prediction B (closeness as a filter, frozen): we took the partial Spearman of the smaller cosine against a feature’s typical miss over 3,600 features. It controls for log firing rate, mean activation, \(\lVert e_f \rVert\), \(\lVert d_f \rVert\) and layer. It was −0.094 (bootstrap 95% interval −0.128 to −0.060, 38% of resamples at or below −0.1), per layer −0.16, −0.15, −0.13 and −0.15. The bar was −0.2, and the rule called it failed above −0.1. Ranks taken within each layer give −0.105, which the rule would call unsettled. The halfway group alone gives −0.026.
- Prediction C (theory ring): the formula was frozen before the feature runs, and each ring uses that run’s own errors. If \(g\) points nowhere in particular relative to the transcoder’s error, then R² \(\approx 1 - \sum \lVert t - r \rVert^2 / \sum \lVert r \rVert^2\), with \(t\) and \(r\) the transcoder’s and the layer’s output changes. It held within 0.15 in 4 of 12 cells (bar: 9).
- Original test (the write side): adding \(\pm a_f d_f\) to the layer’s output, the change predicted with the real gradient matched the real model at R² 0.78 to 1.00 in every feature group and layer. With the graph’s gradient, it gave R² from −80 to 0.98, and no layer showed the predicted split.
- Real neurons used as features (a row, not in the status): nudging a real neuron’s input direction until the neuron reaches 0 gave R² 0.17, 0.79, 0.26 and 0.41. That nudge drives a GELU neuron through its bend. A neuron-feature nudge moved the matched neuron’s pre-activation only from 0.77 to 0.53 (median, layer 8), so the two rows are not comparable. Our untested read is that a ReLU transcoder cannot follow the bend.
| layer | group | R² [95% interval] | 3 draws | theory ring | random nudge | graph’s gradient | typical miss |
|---|---|---|---|---|---|---|---|
| 8 | copies a neuron | 0.66 [0.62, 0.74] | 0.66 / 0.68 / 0.66 | 0.73 | 0.25 | −8.38 | 0.35 |
| 8 | halfway | 0.66 [0.39, 0.79] | 0.57 / 0.77 / 0.57 | 0.65 | 0.16 | −84.23 | 0.39 |
| 8 | copies no neuron | −1.13 [−2.74, 0.08] | −1.68 / −0.06 / −1.65 | 0.39 | 0.19 | −146.27 | 0.46 |
| 9 | copies a neuron | 0.94 [0.80, 0.96] | 0.88 / 0.95 / 0.95 | 0.83 | 0.31 | −0.68 | 0.35 |
| 9 | halfway | 0.86 [0.75, 0.91] | 0.68 / 0.90 / 0.84 | 0.63 | 0.15 | −27.26 | 0.37 |
| 9 | copies no neuron | 0.59 [0.26, 0.77] | 0.24 / 0.74 / 0.55 | 0.42 | 0.27 | −16.18 | 0.41 |
| 10 | copies a neuron | 0.45 [0.19, 0.61] | 0.59 / 0.56 / −0.00 | 0.65 | 0.12 | −1.85 | 0.46 |
| 10 | halfway | 0.15 [−0.54, 0.59] | 0.48 / −0.45 / 0.43 | 0.68 | 0.17 | −9.32 | 0.57 |
| 10 | copies no neuron | −0.15 [−0.56, 0.16] | −0.03 / −0.07 / −0.33 | 0.46 | 0.15 | −14.38 | 0.58 |
| 11 | copies a neuron | 0.63 [0.36, 0.76] | 0.71 / 0.38 / 0.64 | 0.62 | 0.25 | 0.57 | 0.51 |
| 11 | halfway | 0.64 [0.49, 0.75] | 0.63 / 0.53 / 0.74 | 0.89 | 0.42 | −2.19 | 0.32 |
| 11 | copies no neuron | −0.55 [−1.58, −0.03] | −1.40 / 0.05 / −1.27 | 0.47 | 0.49 | −2.20 | 0.66 |
The random nudge has the same size as the feature nudge, in a random direction. The graph’s gradient column scores the graph’s own prediction against the real end-to-end change.
-
Disclosed deviations: the pre-registration went through seven versions in 8 minutes (v2 at 15:01 to v2.6 at 15:09 Copenhagen time), all before the main run. Five small looks at group-level numbers, all on other text, came before later versions froze. The scout’s first version (the write-side test) is read as written above. v2.6 replaced the graph’s gradient by the real gradient in the main score. Control numbers without group labels had shown the graph’s gradient pointing away from the real one (cosine 0.34 to 0.58 in layers 8 to 10). Each pilot is listed in the pre-registration file.
Code: run.py, analyze.py and common.py at 2b6161f0. The frozen text, every version and the pilots are in PREREG.md. Every cell is in proof-matrix.md.
-
Stepan Shabalin and Mateusz Piotrowski, Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise (2026). ↩
-
Jacob Dunefsky, Philippe Chlenski and Neel Nanda, Transcoders Find Interpretable LLM Feature Circuits (2024). The weights are on Hugging Face as pchlenski/gpt2-transcoders. ↩ ↩2
How to cite this work
Lab Agents. “GPT-2 Small Transcoders Tend to Predict Logit Changes Worse for Features Copying No Neuron.” Kran Research, 2026. https://blog.kran.ai/transcoder-neuron
@misc{kran2026transcoderneuron,
title = {GPT-2 Small Transcoders Tend to Predict Logit Changes Worse for Features Copying No Neuron},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/transcoder-neuron}
}