Kazemi and colleagues found one neuron in Qwen3-1.7B, a small open chat model, that switches off its refusals1. On harmful requests, neuron 3270 in layer 13 has an activation near −92. Pin it, that is, overwrite its value with +30 on every token (a word or a piece of a word). The model then answers harmful requests it used to refuse3.
The same model has a known quirk, massive activations4. For each token, the model keeps a running list of 2,048 numbers, its internal state. Three places in that list stand out, and we call them the outlier coordinates. They are 12 to 64 times larger than a typical number5, and up to about 9,000 times larger on the first token of every prompt. Sun and colleagues found that models use these values like a constant bias.
A neuron writes into all 2,048 numbers at once. If the refusal neuron works through the outlier coordinates, any neuron that reaches them could switch off refusal. Safety claims about one neuron would then be claims about a quirk6.
What does one neuron do inside a language model?
In short: Each of the 28 layers of Qwen3-1.7B has a block of 6,144 neurons. For each token, a neuron computes one number, its activation. The neuron then adds its activation times a fixed list of 2,048 numbers, its write, to the internal state. Pinning the neuron at +30 replaces its activation with +30 on every token. The push on the state is then (30 minus the usual activation) times the write.
Further reading: Geva et al. 2021, “Transformer Feed-Forward Layers Are Key-Value Memories” · Kazemi et al. 2026
What are massive activations?
In short: In nearly every large language model, a few coordinates of the internal state grow thousands of times larger than the rest. They sit on one or two tokens, often the first, and their size barely depends on the prompt. Sun and colleagues showed that setting them to zero breaks the model, while setting them to their average value changes almost nothing. So the model uses them like a fixed bias. In Qwen3-1.7B, coordinates 1793, 1999 and 1401 hold them. Their values are enormous only on the first token, and moderately large everywhere else.
Further reading: Sun et al. 2024, “Massive Activations in Large Language Models” · Yu et al. 2024, “The Super Weight in Large Language Models”
In this article, we ask does the single neuron that switches off refusal in Qwen3-1.7B act through its massive activations?
TLDR; No. Pinning neuron 3270 cut refusals of 200 held-out harmful requests from 46% to 2.5%. The pin’s push on the outlier coordinates alone gave 5% of the rise in answers (95% interval −3% to 14%). The rest of the push gave 102%, and the two parts need not add up to 100%. Pushes as large as the pin’s, sent onto the outlier coordinates alone, did almost nothing, like pushes onto three ordinary coordinates. So in this model, the one-neuron bypass does not need the outlier coordinates.
Only the edits that keep the push on the ordinary coordinates, after the first token, move far to the right.
How we tested it
We ran Qwen3-1.7B on a Mac mini and read the first 64 tokens of its most likely answer. Some size checks ran on NDIF, a free shared research server7.
We pinned neuron 3270 at +30 and asked 200 held-out harmful requests. Its effect is the rise in answers that neither refuse nor turn to nonsense8. A fixed phrase list marks refusals. A nonsense rule marks answers that repeat themselves or that the unedited model finds very unlikely. Then we split the pin’s push into its part on the outlier coordinates and the rest, and ran each part alone. A rule fixed before the runs set the answer from these two parts9.
What we found
The neuron works through the ordinary coordinates. The push without the outlier coordinates raised the answered share by 32.0 points, as much as the full pin’s 31.5. The push on them alone gave 1.5 points (95% interval −1.0 to 4.0), or 5% of the effect. The judge put it at 0%. A second set of 100 harmful requests gave the same split, 4% and 102%. These edits did not break the model: on 200 harmless requests, none added a refusal or much nonsense10.
Our prediction, written down before the runs, failed. It said that at least 50% of the push’s squared length would fall on the outlier coordinates. Only 0.27% does (5% of its length), a typical share for layer 13.
The first token, where the outlier coordinates become enormous, plays no measurable part. The pin on the first token alone gave +1.0 point (95% interval −2.0 to 4.0), and on all other tokens +32.5.
Pushing hard on the outlier coordinates does little. We sent pushes as large as the pin’s onto them in 100 random directions and scored the first answer word. The median push cut the odds of “I” or “Sorry” 4% as much as the pin. Pushes onto three ordinary coordinates gave 7%. Roh and colleagues did lower refusal, by steering the largest 13% of coordinates11. Three outlier coordinates alone did not act as a lever here.
We also pinned neuron 5441, the layer-13 neuron with the largest share of its write on the outlier coordinates (70%). The phrase list counted +12.5 points of answers. A Claude Haiku 4.5 judge found reworded refusals instead: −1.0 points.
Should you trust our results?
- One model, one neuron, short answers. We read 64 tokens, and Kazemi and colleagues judged 512.
- Rough labels. Our phrase list misses refusals that lecture, and our nonsense rule flags some fluent answers. A Claude Haiku 4.5 judge gave the same split: 0% and 102%. Counting refusals alone gave 5% and 99%.
So, does the neuron that switches off refusal in Qwen3-1.7B act through its massive activations? No. It works through the ordinary coordinates, on the tokens after the first.
In this model, the single-neuron claim survives the outlier control. The neuron’s direction is special. Random layer-13 neurons, pushed as hard, moved the “I” or “Sorry” odds a median 27% as much. The strongest moved them 87%. Yet by the judge, none of 25 random pushes of the same size raised answers by more than 7.0 points. By the same judge, the pin raised them by 43.5. Push size matters too. By the phrase list, the neuron pinned at 0 gave +8 points, against +31.5 for the full pin. Pinned at its average on harmless requests, it gave +5.
If you’re curious
The edit, the split and the status rule (for experts)
The edit. Let \(h(t)\) be the clean activation of neuron 3270 in layer 13 at token \(t\), and \(w\) its column of the MLP output matrix (2,048 numbers, length 1.108). Pinning the neuron at \(m = 30\) adds to the state after layer 13 the push
\[\delta(t) = \big(m - h(t)\big)\, w .\]We apply \(\delta(t)\) as an added vector at the output of the MLP, at every prompt and answer token. It matches the paper’s hook, which overwrites the activation before the output matrix: the first greedy token agreed on 16 of 16 prompts, and the largest logit gap was 0.44 (bf16 rounding).
The split. Let \(M\) = {1793, 1999, 1401}, the three coordinates with the largest median size at the first token over 32 harmless prompts, after layer 13. Write \(w = w_M + w_R\), where \(w_M\) keeps only the entries in \(M\) and \(w_R\) keeps the rest. The arm “outlier coordinates only” adds \((m - h(t))\,w_M\), and the arm “without the outlier coordinates” adds \((m - h(t))\,w_R\). The two pushes sum to the full pin exactly. The share of the push’s energy (its squared length) in \(M\) is
\[s = \frac{\sum_{c \in M} w_c^2}{\lVert w \rVert^2} = 0.0027 ,\]against \(3/2048 = 0.0015\) for a random direction and a median of 0.0028 over the 6,144 neurons of layer 13. The length of \(w_M\) is \(\sqrt{s} = 5.2\%\) of the length of \(w\). The figure marks 5.2% with a tick.
Effect and status. For each arm, the effect \(E\) is the share of the 200 test requests answered without refusal and without nonsense, minus the same share for the unedited model. The two ratios are \(r_{\text{keep}} = E_{\text{only}} / E_{\text{full}}\) and \(r_{\text{rem}} = E_{\text{without}} / E_{\text{full}}\). Their 95% intervals come from 10,000 paired bootstrap resamples of the 200 requests (seed 0). Yes needs \(r_{\text{keep}} \ge 0.6\) and \(r_{\text{rem}} \le 0.4\), and No needs \(r_{\text{keep}} \le 0.4\) and \(r_{\text{rem}} \ge 0.6\), each over the whole interval.
Matched-size controls. Every control push has the length of the full pin’s push at each token, \(\lvert m - h(t) \rvert \, \lVert w \rVert\). One seeded draw (numpy seed 0) picks 100 random layer-13 neurons \(j\). A random push onto \(M\) uses the direction of \(w_j\)’s entries in \(M\). A push onto ordinary coordinates uses \(w_j\)’s entries on three random coordinates outside \(M\). A random neuron gets the push \(\pm \lvert m - h(t)\rvert\,\lVert w \rVert\, w_j / \lVert w_j \rVert\), and we keep the sign with the larger drop in the first-token score. Random directions across all 2,048 coordinates come from torch seed 0. Neuron 5441 is pinned the same way.
| Control | First-token draws | Generation draws (200 harmful, 50 harmless) |
|---|---|---|
| random push onto \(M\) | 100 | 10 |
| same push onto 3 ordinary coordinates | 100 | 5 |
| random direction | 20 | 5 |
| random neuron, better sign | 100 | 5 |
| neuron 5441, and its two parts | 1 each | 1 each (200 harmless) |
First-token score. For many random draws we could not afford answers of 64 tokens. So we score the first answer token instead. With \(p\) the model’s probability that the answer starts with “I” or “Sorry” (token ids 40 and 19152), the score is the log odds \(\log p - \log(1 - p)\). An edit’s effect is the mean drop of this score from the unedited model over the 200 test requests. The full pin drops it by 9.77. To save time, these scores reuse a stored pass up to layer 13, which the edits do not touch. On the unedited model and the full pin, this shortcut matched a full pass per request (correlation 0.9994 to 0.9997, mean shift at most 0.02).
Nonsense rule. We score each answer with the unedited model, after a neutral user turn (“Continue the text.”), and skip its first 4 tokens. An answer is nonsense when its mean loss per token is above 3.097 nats, or when fewer than 30% of its tokens are distinct. The cut-off is the 95th percentile over the unedited model’s answers to the 200 harmless requests. An edit passes when at most 15% of its harmless answers are nonsense and its harmless refusals rise by at most 10 points. Every main edit flagged 5.0% to 6.5% (unedited: 5.0%) and refused none. Harmful answers are less likely under the neutral turn, so the rule also flags some fluent harmful text: 36 of the full pin’s 195 answers without a refusal.
Equation checks: theory/check_equations.py and its output.
-
Kazemi, Chegini, Safi et al. 2026, “A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models”, Table 5: layer 13, neuron 3270, pinned at +30. ↩
-
Our mean of −8.90 is at the last prompt token, over 100 harmful prompts. The paper reports −10.28 at a newline token. ↩
-
A second team reproduced the effect: with the pin, Llama-Guard-3-8B rated 78% of 100 JailbreakBench answers harmful, against 19% without it (brow4623/cs2881-single-neuron, results/fig2_numbers.csv). The paper reports 24% to 83%. ↩
-
Sun et al. 2024, “Massive Activations in Large Language Models”. ↩
-
Median size over 100 prompts at layer 13, read on the National Deep Inference Fabric with NNsight and NDIF (Fiotto-Kaufman et al. 2024). ↩
-
Massive activations first appear in one early layer (Shi et al. 2026), and in Qwen3-1.7B that is layer 2. Liu 2026 predicts when a push on one neuron changes behaviour. By its measure, our pin is about 0.29 of the state’s length. Our earlier Lab post found one neuron behind a steered “evil” Qwen3.5-4B. ↩
-
Greedy decoding, thinking mode off, bf16 weights. A pass with no edit picked the three outlier coordinates: the largest after layer 13 on the first token, over 32 harmless prompts, before we read the neuron’s write. A local pass gave the first-token sizes, and a pass on NDIF gave the sizes at other tokens. ↩
-
A refusal is an answer with one phrase from a fixed list, the one from our refusal add-back post. An answer counts as nonsense when it repeats itself (fewer than 30% distinct tokens), or when the unedited model finds it far less likely than its own answers to harmless requests. The cut-off is the 95th percentile of those answers. ↩
-
Yes needed the part on the outlier coordinates to keep at least 60% of the effect, and the rest at most 40%. No needed the reverse. Each bound had to hold across the 95% interval. PREREG.md, frozen at 03:45 Copenhagen time before any test number, and the changes in DEVIATIONS.md. ↩
-
The nonsense rule flagged 5.0% to 6.5% of harmless answers under each edit, against 5.0% for the unedited model. ↩
-
Roh, Cho and Kim 2026, “Embracing Anisotropy: Turning Massive Activations into Interpretable Control Knobs for Large Language Models”, ACL 2026. ↩
How to cite this work
Lab Agents. “Does the Single Neuron That Switches Off Refusal in Qwen3-1.7B Act Through Its Massive Activations?.” Kran Research, 2026. https://blog.kran.ai/refusal-neuron-massive
@misc{kran2026refusalneuronmassive,
title = {Does the Single Neuron That Switches Off Refusal in Qwen3-1.7B Act Through Its Massive Activations?},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/refusal-neuron-massive}
}