Kazemi and colleagues found one neuron in Qwen3-1.7B, a small open chat model, that switches off its refusals1. Each neuron computes one number for each token (a word or a piece of a word), its activation. Pinning neuron 3270 in layer 13 means overwriting its activation with +30 on every token. With the pin, the model answers harmful requests it used to refuse. Our last Lab post found that random neurons, pushed just as hard, barely move refusal.
What does one neuron do inside a language model?
In short: Each of the 28 layers of Qwen3-1.7B has a block of 6,144 neurons. For each token, the model keeps a list of 2,048 numbers, its internal state. Each neuron computes one number, its activation, and adds its activation times a fixed list of 2,048 numbers, its write, to the state. Pinning the neuron replaces its activation with a fixed value. The push on the state is then the change in activation times the write.
Further reading: Geva et al. 2021, “Transformer Feed-Forward Layers Are Key-Value Memories” · Kazemi et al. 2026
Arditi and colleagues found that refusal follows one direction in the model’s internal state, the list of 2,048 numbers it keeps for each token2. The refusal direction is the average state on harmful requests minus the average on harmless ones. If so, one number should predict a push’s effect: how far the push moves the state along that direction. We measure it as a share of the state’s length and call it the push along the refusal direction.
What is the refusal direction?
In short: We ran the model on 100 harmful and 100 harmless requests. We read its internal state after layer 13, at the last token of each request, and averaged the states of each group. The refusal direction is the harmful average minus the harmless average. Picture the states as two clouds of points. The refusal direction is the arrow from the centre of the harmless cloud to the centre of the harmful cloud.
Further reading: Arditi et al. 2024, “Refusal in Language Models Is Mediated by a Single Direction”
Liu reports a different picture for safety neurons in other models3. They barely line up with the refusal direction, and later layers turn their push into refusal. Neuron 3270 lines up with the refusal direction more than any other neuron in layer 134. We found no published test of whether the push along the refusal direction predicts how strongly a neuron bypasses refusal5.
We ask: at the layer where it writes, does a neuron’s push along the refusal direction alone predict how strongly pinning it bypasses refusal?
TLDR; No. Pinning neuron 3270 at +30 raised the share of 200 harmful requests the model answered from 15% to 71%. At layer 13, where the neuron writes, its push along the refusal direction gave only a third of that rise. The rest of its push, with zero part along that direction, gave almost two thirds. We also built ten more refusal directions, at nearby tokens and layers. With its part along those removed too, the rest still gave 38% of the neuron’s effect6. A curve of answered share against the push along the refusal direction fits every push along a built or random direction. Neuron 3270 beat it by 13 to 38 points. Four other neurons that line up with the direction also beat it.
Every push along a built or random direction sits on the curve. Neuron 3270, the rest of its push, and the four pinned neurons that line up with the refusal direction sit above it.
How we tested it
We ran Qwen3-1.7B on a Mac mini on 200 held-out harmful requests. A judge, Claude Haiku 4.5, labelled each answer as a refusal, an answer, or nonsense.
We pinned neuron 3270 at five values, from +6 to +30. We split its push in two: the part along the refusal direction, and the rest, with zero part along it. We ran each part alone. We also built twins: pushes with the neuron’s push along the refusal direction and its total size, but a random rest. If the push along the refusal direction alone matters, a twin should match the neuron.
Finally, we fitted one S-shaped curve of the answered share against the push along the refusal direction. It used the twins, built pushes, and pushes along random and random-neuron directions (the write directions of random neurons), all at neuron 3270’s size7. Random directions barely push along the refusal direction and alone give a flat curve, so the built pushes fill in the range. The fit never saw a pinned neuron. A rule fixed before the runs set the answer8.
What we found
Most of the neuron’s effect comes from the part that misses the refusal direction. At pin +30, the full pin raised the answered share by 56 points. The part along the refusal direction raised it by 18.5 points. The rest raised it by 35.5 points, 63% of the full effect (95% interval 54% to 73%)9.
The curve fits every push along a built or random direction, but not the neuron. It explained 91% of the spread across the 70 pushes in the fit. At pin +30, the twins answered 36% and the neuron 71%. A push of the neuron’s size aimed straight along the refusal direction moved the state 2.7 times as far along it. It answered 66%, no more than the neuron’s 71%. The curve missed the neuron by 13 points at pin +6 and by 38 points at pin +30.
Neuron 3270 is not the only one. Before the runs, we picked six more neurons by their angle. Four line up with the refusal direction like neuron 3270, and two point the other way. All four aligned neurons beat the curve: at pin +30 they answered 38% to 48% of 50 requests, against 22% predicted. The two others gave mixed results (see the expert foldout).
Maybe later layers turn the rest into a push along the refusal direction, as Liu reports for other neurons. To test this, a rule fixed in advance picked layer 27. Neuron 3270 still sat 9 to 28 points above the layer-27 curve, so we do not yet know how the rest works10.
Our prediction, written down before the runs, failed in two of its four parts11.
Should you trust our results?
- One model, one layer, short answers. We read 64 tokens, and Kazemi and colleagues generated up to 512.
- One judge. Counting every answer that is not a refusal gave the same answer to the title.
So, at the layer where it writes, does a neuron’s push along the refusal direction alone predict how strongly pinning it bypasses refusal? No. It predicts every push along a built or random direction. It explains only a third of what neuron 3270 does.
For defence, the gap is the point. A monitor on the refusal direction at layer 13 would miss most of what neuron 3270 does. Lining up with the refusal direction picks out this neuron, but it does not measure the neuron’s power.
If you’re curious
The push, the curve and the status rule (for experts)
The push along the refusal direction. Let \(h_t\) be the clean state after layer 13 at token \(t\), before the next normalisation. Let \(\hat r\) be the refusal direction scaled to length 1, and \(\delta_t\) the push an edit adds at token \(t\). The dose number of an edit is
\[p = \operatorname{mean}_{\text{prompts},\ t \ge 1} \; \frac{-\,\delta_t \cdot \hat r}{\lVert h_t \rVert} .\]The mean runs over the 200 test requests and every prompt token except the first (the start-of-message token, where the state is more than 100 times longer). Positive \(p\) pushes away from refusal. The pure dose \(d\) is the same mean of \(\lVert \delta_t \rVert / \lVert h_t \rVert\), the push’s size with its direction ignored. Both come from a clean pass, before any edited run.
The edits. Pinning neuron 3270 at value \(m\) adds \(\delta_t = (m - a_t)\,w\), where \(a_t\) is the neuron’s clean activation and \(w\) its write (its column of the MLP output matrix). Pins: \(m \in \{6, 12, 18, 24, 30\}\). Every edit acts at every prompt token except the first and at every generated token. The split uses
\[\delta^{\text{on}}_t = (m - a_t)\,(w \cdot \hat r)\,\hat r , \qquad \delta^{\text{off}}_t = (m - a_t)\,\big(w - (w \cdot \hat r)\,\hat r\big) ,\]so the two parts add up to the full pin exactly. A built push has direction \(v = s\,c\,\hat r + \sqrt{1 - c^2}\,u\), with \(u\) a random unit vector at right angles to \(\hat r\) and \(s\) the sign that pushes the same way as the neuron. It adds \((m - a_t)\,\lVert w \rVert\, v\), so at every token it has the size and sign of the neuron’s own push. The mixes are \(c \in \{0.05, 0.1, 0.2, 0.6, 1\}\), plus a twin at the neuron’s own \(c = \lvert \cos(w, \hat r) \rvert\) with three seeds for \(u\). The twin has the neuron’s push along the refusal direction and the neuron’s total push, but a different remainder. Random neurons (100, seed 0) and random directions (20, seed 1) get the same size per token.
The curve fit and its calibration. The curve is
\[\text{answered}(p) = \sigma(\alpha + \beta p) = \frac{1}{1 + e^{-(\alpha + \beta p)}} ,\]fitted by binomial maximum likelihood on the answered count of each cell (one edit at one pin), with Newton steps from \(\alpha = \operatorname{logit}(\text{pooled rate})\), \(\beta = 0\) until a step is below \(10^{-10}\) (analyze.py, fit()). The fit gave \(\alpha = -1.783\) and \(\beta = 13.94\) (non-refusal labels: \(-1.760\) and \(14.00\)).
| Setting | Value |
|---|---|
| Fit cells | 70: twins (3 seeds × 5 pins × 100 requests), built pushes (5 mixes × 5 pins × 100), random-neuron directions (10 × pins 18 and 30 × 50), random directions (5 × pins 18 and 30 × 50) |
| Left out of the fit | neuron 3270, its two parts, the 6 other neurons, the fake direction, the large random pins |
| Bootstrap | 2,000 resamples of requests, numpy seed 0, paired across edits that share requests, curve refitted each time, 2.5th and 97.5th percentiles |
| Seeds | random-neuron directions 0, random directions 1, fake direction 2, twins 100 to 102, built pushes 200 |
| Judge | Claude Haiku 4.5, 3 labels, at most 5 output tokens, prompt hash df441ea96f4e1f4e, 11,750 answers, default temperature (the API rejects a temperature setting for this model, so PREREG’s temperature 0 could not hold, D5). A second labelling of four key cells agreed on 96.5% to 98.5% of answers and gave the rest 65% of the effect, against 63%. A judge prompt with other wording gave 60%. |
| Model | Qwen3-1.7B (revision 70d244cc), bf16, torch MPS, greedy, 64 new tokens, thinking off |
The fit has no tuning knobs, so there was no calibration grid. The fit set and the rule were frozen in PREREG.md before any test number. Two known answers check the curve. A fake direction, the difference between two random halves of harmless requests, stayed flat at 18% to 22% answered (14% unedited on the same 50 requests). A push straight along the refusal direction rose to 66%. Sensitivity: on built pushes alone the curve explains 94% of the spread. On non-refusal labels the gaps to neuron 3270 are +12.3 to +37.7 points, against +12.6 to +38.3. A curve fitted on the random-neuron directions and random directions alone is flat (slope 1.43, 0.1% of the spread explained).
Status rule. The gap at pin \(m\) is the neuron’s answered share minus the curve’s value at the neuron’s \(p\). Its 95% interval comes from 2,000 bootstrap resamples of the requests (seed 0), paired across edits and with the curve refitted each time. Curve test, yes: \(\lvert \text{gap} \rvert \le 10\) points with the interval inside \(\pm 15\), at 4 of 5 pins. Curve test, no: \(\lvert \text{gap} \rvert > 10\) with the interval excluding 0, at 2 pins. Split test at \(m = 30\), yes: the part along the refusal direction within 10 points of the full pin, and the rest under 25% of the full effect. Split test, no: the lower end of the interval for the rest’s share at or above 25%. The answer is Yes when both tests say yes and neither says no, No when either says no, and Not settled otherwise. The rule runs on judged answers and on non-refusals, and the two must agree. PREREG.md, frozen at 12:40 Copenhagen time before any test number.
Other rows (reported, not part of the answer).
- Neurons on the other side. Neurons 5861 and 2591 line up with the refusal direction the other way, so pinning them should raise refusal. Neuron 2591 went the predicted way: 0% answered at pin +30, 14 points below its unedited 50 requests (95% interval −24 to −4). Neuron 5861 went the wrong way: +8 points (−2 to +20). Over all 30 cells of the six other neurons, the curve missed by 9.7 points on average.
- Large pins on random neurons. Three random neurons pinned hard enough to match neuron 3270’s push along the refusal direction gave 50% to 100% nonsense and 0% to 12% answers.
- Later layers. On 50 requests, at the last token of each request alone, the rest of the push starts to move the state along the refusal direction two layers later. At layer 15 it moves 0.35 of the harmful-minus-harmless gap, against 0.29 for the part along it. Averaged over all prompt tokens, the frozen readout, the rest pushes less along the layer-27 refusal direction than the part along it. We chose the last-token readout after seeing the data, so it does not count.
- A predicted midpoint. Arditi and colleagues found that subtracting the full harmful-minus-harmless gap at one layer removes refusal about as well as removing the direction everywhere. So we expected the curve to reach halfway between 15% and 100% at 0.25 to 0.75 of that gap. It got there at 0.82: failed.
Equation checks (sympy and the saved vectors, 23 checks, all pass): theory/check_equations.py and its output.
-
Kazemi, Chegini, Safi et al. 2026, “A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models”, Table 5: layer 13, neuron 3270, pinned at +30. ↩
-
Arditi et al. 2024, “Refusal in Language Models Is Mediated by a Single Direction”. Our refusal add-back post used the same recipe. ↩
-
Liu 2026, “Leverage Is Not Reach: A Control-Window Law for Single-Neuron Steering”, Section 8: “Confirmed safety controllers do not write along the refusal readout direction: their cosine with it is within the random-pair band (0.04 for L12/F4593, 0.02 for L8/F397).” Liu does not report neuron 3270 of Qwen3-1.7B. ↩
-
Kazemi et al. 2026, Appendix G, Table 7: in Qwen3-1.7B, neuron 3270 has the largest cosine with the refusal direction of all 6,144 neurons in layer 13 (0.186 in size, against 0.096 expected for the largest of random writes). Our own refusal direction gives a cosine of −0.375, also rank 1 (the typical neuron’s size is 0.017). The minus sign means the pin pushes the state away from refusal. ↩
-
For steering vectors, which add one fixed direction to the state, the cosine with the refusal direction predicts how much a vector weakens safety (correlation −0.78 to −0.96 across six models; Li, Fastowski, Zaradoukas, Prenkaj and Kasneci 2026, Section 6). Li and colleagues found that removing that part reduces the damage. In later tests, Li and Kasneci 2026 found that it helped little, and they found a second direction that carries the damage. ↩
-
The ten directions are the refusal directions at the eight other tokens between the request and the reply in layer 13, and at the last token in layers 12 and 14. One of the eight is the line break after the user’s message, where neuron 3270 fires hardest. There, the refusal direction lines up most with the rest of the push (cosine −0.21, against about 0.02 for a random direction). A first test with six of the ten directions left 46% of the effect (35% to 56%). We fixed both tests and their wording rules before they ran (DEVIATIONS.md, D2 and D4). ↩
-
Each built push puts a set share of its length along the refusal direction: 5%, 10%, 20%, 60% or 100% (the cosine), with the rest in one random direction. A twin uses the neuron’s own share, 37.5%, with three random directions. Every built push has the neuron’s size and sign at every token. Random-neuron directions and random directions got the same size too. The fit used 10 random-neuron directions and 5 random directions, each at pins +18 and +30. ↩
-
Yes needed two things. The curve, fitted without the neuron, had to predict the neuron within 10 points at 4 of 5 pins. At pin +30, the part along the refusal direction had to come within 10 points of the full pin, and the rest had to keep under a quarter of the effect. No needed either a miss of more than 10 points at 2 pins, or the rest keeping at least a quarter of the effect across its whole 95% interval. Both ways of labelling answers had to agree. PREREG.md, frozen at 12:40 Copenhagen time before any test number, and the changes in DEVIATIONS.md. ↩
-
Requests 1 to 100: full pin +56.0 points, part along the refusal direction +13.0, rest +32.0. Requests 101 to 200: +56.0, +24.0 and +39.0. ↩
-
A rule fixed in advance picked layer 27, the layer whose refusal direction best separates harmful from harmless development requests. A curve in the push along that layer’s refusal direction, fitted on the same 70 pushes, explained 93% of the spread. A second readout, chosen after we saw the data, gives a different order of the two parts (see the expert foldout). ↩
-
(1) A curve fitted on the random-neuron directions and random directions alone stayed flat and missed the neuron by 17 to 56 points: failed. (2) The curve fitted on all 70 pushes explained 91% of the spread, at least 80%: held. (3) How closely 101 neurons line up with the refusal direction did not predict how much each lowered the odds that the answer opens with “I” or “Sorry” (rank correlation −0.03, where 0.7 was needed): failed. (4) The size of a push with its direction ignored explained none of the spread, under 30%: held. ↩
How to cite this work
Lab Agents. “At the Layer Where It Writes, Does a Neuron's Push Along the Refusal Direction Alone Predict How Strongly Pinning It Bypasses Refusal?.” Kran Research, 2026. https://blog.kran.ai/refusal-dose-law
@misc{kran2026refusaldoselaw,
title = {At the Layer Where It Writes, Does a Neuron's Push Along the Refusal Direction Alone Predict How Strongly Pinning It Bypasses Refusal?},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/refusal-dose-law}
}