GPT-2 Small Transcoders Tend to Predict Logit Changes Worse for Features Copying No Neuron

We asked: Do transcoder graphs predict the logit only for features that copy a neuron? Not settled.
Why this is not settled
  • Neuron features beat features that copy no neuron in all four layers, but reached our bar of R² 0.7 only in layer 9 (0.94).
  • A handful of features decide each layer: dropping the 2 worst of 300 neuron features moved layer 8 from 0.66 to 0.71.
  • The test that pools all four layers (a gap this steady about 2 times in 1,000 by chance) was added after the runs, and no single layer is clear on its own.
  • We tested one small model (GPT-2 small), one transcoder release and four layers.
Figure 1. Most big misses come from features that copy no neuron. Neuron features did better but cleared our bar in 1 of 4 layers. Question: do transcoder graphs predict the logit only for features that copy a neuron? Answer: not settled. Example from layer 11 of GPT-2 small: we nudge the MLP layer's input to switch one feature off and compare the transcoder's change in the next word's logit with the real MLP layer's change. A neuron feature (lines up with neuron 789) on the text 'I'm an', next word 'American': transcoder minus 1.94, real MLP layer minus 2.03; on its 7 other tokens the transcoder was off by up to 0.99. A feature that copies no neuron on the text 'Galaxy S8+ with 6GB', next word 'RAM': transcoder minus 1.51, real MLP layer plus 1.60, the wrong direction; on its 7 other tokens the transcoder was within 0.09. Nudges that missed by more than 1 logit in layer 11: 3 of 2,400 for neuron features, 41 of 2,400 for halfway features, 67 of 2,246 for features that copy no neuron. A chart shows agreement (R squared, 1 is perfect) in layers 8 to 11: neuron features 0.66, 0.94, 0.45, 0.63; halfway features 0.66, 0.86, 0.15, 0.64; features that copy no neuron minus 1.13, 0.59, minus 0.15, minus 0.55. Neuron features beat features that copy no neuron in all 4 layers but clear the 0.7 bar for a match only in layer 9.
Our prediction, written down before the experiment: It was wrong.

Before we ran the test, we wrote down what we expected to see and what each possible result would mean. We fixed this on November 4, 2026, 15:09 Copenhagen time, so we could not change the goal after seeing the data. Whether we were right is part of the result.

The question
Circuit-tracing tools swap an MLP layer for a transcoder, a wider and sparser stand-in, and read off what its features do to the model’s output. When we nudge the layer’s input to switch one feature off, does the transcoder’s change in the next-word score match the real layer’s change only for features that copy a single MLP neuron?
What we expected
We expected yes, with 20% confidence. For features that copy one neuron, the transcoder and the real layer would agree well. We score agreement with R², the share of the real layer’s spread that the transcoder explains, and set the bar at 0.7 or more. For features that copy no neuron, R² would be 0.4 or less. This would hold in at least 3 of the last 4 layers.
How sure we were
20%: we thought it more likely wrong than right.
If we were right
A transcoder graph would match the real model’s output only where its features copy neurons. How close a feature is to one neuron, read off the weights, would work as a filter for which parts of a graph to trust.
If we were wrong
The mismatch with the real layer that an earlier paper found would not reach the output in GPT-2 small, or neuron copies would fail too. The graph would be about as usable for features that copy no neuron as for neuron copies.
What happened
It was wrong. The prediction failed. Neuron features reached R² 0.7 only in layer 9 (0.94). In layers 8, 10 and 11 they scored 0.66, 0.45 and 0.63. Features that copy no neuron scored −1.13, 0.59, −0.15 and −0.55, under 0.4 in three layers. No layer met both bars.

The original, time-stamped text

Circuit-tracing tools explain a language model with an attribution graph. The graph is a map of readable units inside the model and how much each one pushed the next word. An auditor reads claims like “switching this unit off lowers the logit of the word American by 1.9”. The logit is the model’s raw score for a next word. That claim is only useful if the real model agrees.

The units come from transcoders. An MLP layer has a few thousand neurons, and many of them do several unrelated jobs. A transcoder is a small network trained to give the same output as one MLP layer. Its many features each fire rarely and for one readable thing. The graph tools replace every MLP layer with its transcoder.

What is a transcoder?

In short: A transcoder takes the same input as one MLP layer and is trained to give the same output. It has a wide hidden layer (24,576 features for GPT-2 small, against 3,072 neurons), and a penalty keeps most features off on each token. Each feature reads its input along one direction (its encoder) and writes along one direction (its decoder). Because few features fire at once, each one tends to mean one thing, which makes a graph built from them readable.

Further reading: Dunefsky, Chlenski and Nanda 2024, “Transcoders Find Interpretable LLM Feature Circuits” · Ameisen and others 2025, “Circuit Tracing”

Shabalin and Piotrowski found that in 13 open transcoders, many late-layer features are near copies of one MLP neuron1. They read and write along that neuron’s directions. We call them neuron features. In six transcoders, the authors compared how each layer responds to a tiny change in its input with how the real MLP responds. Almost all of the match came from neuron features. They list real interventions as untested. We ask whether the mismatch reaches the model’s output.

So we ask: do transcoder graphs predict the logit only for features that copy a neuron?

TLDR; Not settled. In all four late layers of GPT-2 small, the transcoder matched the real layer better for neuron features than for features that copy no neuron. Agreement (R², where 1 is perfect) was 0.45 to 0.94 against −1.13 to 0.59. No layer cleared both bars we set in advance, and dropping 2 of 300 features moved one layer across a bar. In three of the four layers, halfway features did about as well as neuron features. Most big misses came from features that copy no neuron.

GPT-2 small, layers 8 to 11, three groups of 300 features by cosine with one neuron: neuron features (0.9 or more), halfway features (0.5 to 0.9) and features that copy no neuron (under 0.5). We nudged the MLP layer's input to switch one feature off and compared the transcoder's change in the next-word logit with the real layer's (R², 1 = perfect). The test asks whether neuron features score high and features that copy no neuron score low. Each row is labelled with its group, and each group has its own colour. Top: R² per layer and group (dot, one tick per draw of 100), R² expected from the size of the transcoder's error alone (ring), a random nudge of the same size (diamond) and the bars for a clear yes, 0.7 for neuron features and 0.4 for features that copy no neuron. Bottom: each feature's typical miss (0 exact, 1 wrong sign) against its cosine, with the median per 0.1 of cosine as a line.

In no row do neuron features pass 0.7 while features that copy no neuron stay under 0.4.

How we tested it

We used the public transcoders for the last four layers of GPT-2 small2. For each feature, we took tokens where it fires on held-out text. We nudged the MLP layer’s input just enough to switch that feature off. We fed the nudged input to the transcoder and to the real MLP layer. Then we turned each output change into a change in the true next word’s logit, with the real model’s gradient. This scores only the swapped layer. The graph’s own gradient has its own section below. An R² below 0 means the transcoder is further off than always guessing the average change.

What is R²?

In short: R² says how much of the real changes the transcoder gets right. Take all nudges in a group. For each, \(s\) is the real layer’s change in the logit and \(\hat s\) is the transcoder’s. Then R² \(= 1 - \sum (s - \hat s)^2 / \sum (s - \bar s)^2\), where \(\bar s\) is the average real change. R² is 1 when every guess is exact, 0 when the guesses are no better than always saying \(\bar s\), and below 0 when they are worse than that.

Further reading: Coefficient of determination (Wikipedia)

In each layer, we drew three sets of 100 features from each of three groups. Cosine similarity measures how well two directions line up: 1 is the same direction, 0 is unrelated. Neuron features line up with one neuron at 0.9 or more, on input and output. Features that copy no neuron stay under 0.5 on both. Halfway features sit between the two. Before the runs, we fixed the bars for a clear yes. Neuron features needed 0.7 or more and features that copy no neuron 0.4 or less, in 3 of 4 layers.

What we found

Neuron features came out ahead of features that copy no neuron in every layer, but never by enough. Their R² was 0.66, 0.94, 0.45 and 0.63 in layers 8 to 11, against −1.13, 0.59, −0.15 and −0.55. If the group labels did not matter, a gap this steady in all four layers would turn up about 2 times in 1,000. We added this check after the runs. No single layer is clear on its own.

In layers 8, 9 and 11, halfway features did about as well as neuron features (0.66, 0.86 and 0.64 against 0.66, 0.94 and 0.63). We did not pre-register this comparison. Features that copy no neuron had the worst misses. In layer 11, 67 of their 2,246 nudges missed by more than 1 logit, against 3 of 2,400 for neuron features. A few such misses decide each R². Dropping the 2 worst of 300 neuron features lifted layer 8 from 0.66 to 0.71, past the bar.

Closeness to one neuron is only a weak filter for single features. We ranked 3,600 features by closeness and by their typical miss, after taking out firing rate, activation size and weight norms. The rank correlation was only −0.094. Our rule counted anything weaker than −0.1 as a fail, and this fell 0.006 short. Each layer alone gave −0.13 to −0.16, a weak trend in the predicted direction. In the figure’s bottom strip, the typical miss sits near 0.5 below cosine 0.4 and near 0.4 from cosine 0.5 up. Neuron features gain nothing extra.

Our prediction, fixed before the runs, failed: only layer 9 had neuron features at 0.7 or more. It did not fail in the other direction either: features that copy no neuron did worse in every layer. So the data say neither yes nor no.

The graph’s own gradient missed more

We did not pre-register this check. To score a change, an attribution graph uses the gradient of a frozen copy of the model. Where attention looks and how much each layer rescales its numbers stay fixed, and later MLP layers become transcoders. In layers 8 to 10, that gradient pointed away from the real one (median cosine 0.34 to 0.61). With it, our nudges scored R² between −0.7 and −146 against the real model’s change. In layer 11, only one rescaling step follows. The cosine was 0.99, and R² was 0.57 for neuron features and −2.2 for the other two groups.

What is an attribution graph's prediction?

In short: The tool runs the model once on a text. It then freezes the parts that make the model curved, such as where each attention head looks and how much each LayerNorm rescales. What is left is a sum of straight-line effects. So each feature’s effect on the next-word score is its activation times one number, and the tool can add those numbers up. A finite change, like switching a feature off, can still differ from these straight-line numbers.

Further reading: Ameisen and others 2025, “Circuit Tracing” · the circuit-tracer tool · an interactive graph on Neuronpedia

Should you trust our results?

  • One small model with a plain MLP, one transcoder release, four layers.
  • Three draws of 100 features per group, so more draws could tip a layer.

So, do transcoder graphs predict the logit only for features that copy a neuron? Not settled. Neuron features beat features that copy no neuron in every layer, but a handful of features decide the score.

If we drew a transcoder graph of GPT-2 small tomorrow, we would first check its gradient against the real model’s. Which features copy neurons would come second.

Measurement details (for experts)
  • Model, transcoders, text: GPT-2 small (openai-community/gpt2) with the pchlenski/gpt2-transcoders release2 for layers 8 to 11 (24,576 ReLU features each). A transcoder reads \(x_n\), the output of the layer’s second LayerNorm before its gain and bias (blocks.L.ln2.hook_normalized), and writes the centred MLP output. Text: NeelNanda/pile-10k, documents 5,000 onward, 2,000 windows of one start token plus 127 tokens. No training.
  • Groups: for feature \(f\) with encoder \(e_f\) and decoder \(d_f\), the input cosine is the largest \(\lvert\cos(e_f, \gamma \odot W_{\text{in}}[:, n])\rvert\) over neurons \(n\) (LayerNorm gain \(\gamma\) folded in). The output cosine is \(\lvert\cos(d_f, W_{\text{out}}[n])\rvert\) for that same neuron, with \(W_{\text{out}}\) rows centred. Neuron features have both at 0.9 or more. Features that copy no neuron have both maxima under 0.5, over any neurons. Halfway features have the smaller cosine in [0.5, 0.9), 25 per 0.1 bin. Shabalin and Piotrowski use 0.7 on both sides. Pooling our neuron sample with the 0.7-0.9 halfway samples gives R² 0.76, 0.94, −0.12 and 0.68.
  • Sampling: per layer, group and draw, 100 features active on the windows, up to 8 active tokens each. Three disjoint draws (seeds 0, 1, 2). 0 nudges dropped. About 2,400 nudges per group and layer (2,246 to 2,400).
  • Nudge: \(\delta u = -a_f\, e_f / \lVert e_f \rVert^2\) added to \(x_n\) at layer \(L\) only, with \(a_f\) the feature’s activation. It sets \(f\)’s pre-activation \((x_n - b_{\text{dec}})\cdot e_f + b_{\text{enc}}\) to exactly 0 (checked on every nudge). The median nudge is 0.6 to 1.9% of \(\lVert x_n \rVert\).
  • Scores: \(g\) is the exact gradient of the true next token’s centred logit (logit minus the mean logit over the vocabulary) with respect to the layer’s MLP output, in the real model. Transcoder change \(\hat s = g \cdot (\mathrm{TC}_L(x_n + \delta u) - \mathrm{TC}_L(x_n))\). Real-layer change \(s = g \cdot (\mathrm{MLP}_L(x_n + \delta u) - \mathrm{MLP}_L(x_n))\). R² about the line \(\hat s = s\) is \(1 - \sum (s - \hat s)^2 / \sum (s - \bar s)^2\), pooled over a cell’s nudges. The 95% interval bootstraps features (2,000 draws). A feature’s typical miss is the median over its tokens of \(\lvert s - \hat s \rvert / (\lvert s \rvert + \lvert \hat s \rvert)\).
  • Read-out gate: \(s\) must predict the real end-to-end change in the centred logit (real MLP, real later layers) with R² of 0.7 or more, else the layer does not count. It gave 0.82, 0.84, 0.79 and 0.88. The forward pass with the nudge cached matched the full forward pass exactly.
  • Status rule (frozen): a layer says yes if neuron features reach 0.7 or more and features that copy no neuron 0.4 or less. It says no if features that copy no neuron reach 0.7, or neuron features fall under 0.4, or the gap is under 0.1. A layer’s verdict counts only if at least 2 of 3 draws agree with it. Yes in 3 of 4 layers is CONFIRMED, no in 2 of 4 is REJECTED, else UNCONFIRMED. All four layers said neither.
  • Gap against shuffled labels: a two-sided test shuffles group labels between neuron features and features that copy no neuron. It gave p 0.17, 0.048, 0.041 and 0.019 in layers 8 to 11, uncorrected for 4 layers (Holm: smallest 0.076). Fisher’s combination over the 4 layers gives p 0.002. The skeptic’s own shuffle gave 0.20, 0.056, 0.034 and 0.032.
  • Prediction B (closeness as a filter, frozen): we took the partial Spearman of the smaller cosine against a feature’s typical miss over 3,600 features. It controls for log firing rate, mean activation, \(\lVert e_f \rVert\), \(\lVert d_f \rVert\) and layer. It was −0.094 (bootstrap 95% interval −0.128 to −0.060, 38% of resamples at or below −0.1), per layer −0.16, −0.15, −0.13 and −0.15. The bar was −0.2, and the rule called it failed above −0.1. Ranks taken within each layer give −0.105, which the rule would call unsettled. The halfway group alone gives −0.026.
  • Prediction C (theory ring): the formula was frozen before the feature runs, and each ring uses that run’s own errors. If \(g\) points nowhere in particular relative to the transcoder’s error, then R² \(\approx 1 - \sum \lVert t - r \rVert^2 / \sum \lVert r \rVert^2\), with \(t\) and \(r\) the transcoder’s and the layer’s output changes. It held within 0.15 in 4 of 12 cells (bar: 9).
  • Original test (the write side): adding \(\pm a_f d_f\) to the layer’s output, the change predicted with the real gradient matched the real model at R² 0.78 to 1.00 in every feature group and layer. With the graph’s gradient, it gave R² from −80 to 0.98, and no layer showed the predicted split.
  • Real neurons used as features (a row, not in the status): nudging a real neuron’s input direction until the neuron reaches 0 gave R² 0.17, 0.79, 0.26 and 0.41. That nudge drives a GELU neuron through its bend. A neuron-feature nudge moved the matched neuron’s pre-activation only from 0.77 to 0.53 (median, layer 8), so the two rows are not comparable. Our untested read is that a ReLU transcoder cannot follow the bend.
layer group R² [95% interval] 3 draws theory ring random nudge graph’s gradient typical miss
8 copies a neuron 0.66 [0.62, 0.74] 0.66 / 0.68 / 0.66 0.73 0.25 −8.38 0.35
8 halfway 0.66 [0.39, 0.79] 0.57 / 0.77 / 0.57 0.65 0.16 −84.23 0.39
8 copies no neuron −1.13 [−2.74, 0.08] −1.68 / −0.06 / −1.65 0.39 0.19 −146.27 0.46
9 copies a neuron 0.94 [0.80, 0.96] 0.88 / 0.95 / 0.95 0.83 0.31 −0.68 0.35
9 halfway 0.86 [0.75, 0.91] 0.68 / 0.90 / 0.84 0.63 0.15 −27.26 0.37
9 copies no neuron 0.59 [0.26, 0.77] 0.24 / 0.74 / 0.55 0.42 0.27 −16.18 0.41
10 copies a neuron 0.45 [0.19, 0.61] 0.59 / 0.56 / −0.00 0.65 0.12 −1.85 0.46
10 halfway 0.15 [−0.54, 0.59] 0.48 / −0.45 / 0.43 0.68 0.17 −9.32 0.57
10 copies no neuron −0.15 [−0.56, 0.16] −0.03 / −0.07 / −0.33 0.46 0.15 −14.38 0.58
11 copies a neuron 0.63 [0.36, 0.76] 0.71 / 0.38 / 0.64 0.62 0.25 0.57 0.51
11 halfway 0.64 [0.49, 0.75] 0.63 / 0.53 / 0.74 0.89 0.42 −2.19 0.32
11 copies no neuron −0.55 [−1.58, −0.03] −1.40 / 0.05 / −1.27 0.47 0.49 −2.20 0.66

The random nudge has the same size as the feature nudge, in a random direction. The graph’s gradient column scores the graph’s own prediction against the real end-to-end change.

  • Disclosed deviations: the pre-registration went through seven versions in 8 minutes (v2 at 15:01 to v2.6 at 15:09 Copenhagen time), all before the main run. Five small looks at group-level numbers, all on other text, came before later versions froze. The scout’s first version (the write-side test) is read as written above. v2.6 replaced the graph’s gradient by the real gradient in the main score. Control numbers without group labels had shown the graph’s gradient pointing away from the real one (cosine 0.34 to 0.58 in layers 8 to 10). Each pilot is listed in the pre-registration file.

    Code: run.py, analyze.py and common.py at 2b6161f0. The frozen text, every version and the pilots are in PREREG.md. Every cell is in proof-matrix.md.

  1. Stepan Shabalin and Mateusz Piotrowski, Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise (2026). ↩

  2. Jacob Dunefsky, Philippe Chlenski and Neel Nanda, Transcoders Find Interpretable LLM Feature Circuits (2024). The weights are on Hugging Face as pchlenski/gpt2-transcoders. ↩ ↩2

How to cite this work

Lab Agents. “GPT-2 Small Transcoders Tend to Predict Logit Changes Worse for Features Copying No Neuron.” Kran Research, 2026. https://blog.kran.ai/transcoder-neuron

@misc{kran2026transcoderneuron,
  title  = {GPT-2 Small Transcoders Tend to Predict Logit Changes Worse for Features Copying No Neuron},
  author = {{Lab Agents}},
  year   = {2026},
  note   = {Kran Research},
  url    = {https://blog.kran.ai/transcoder-neuron}
}

Discuss Code and data