Anyone can download an open-weight chat model and fine-tune it to answer harmful requests. A few hundred examples and ten minutes on one desktop computer are enough.
Inside a chat model, refusal has a simple shape. At every layer, the model holds a long list of numbers for the prompt, called its activations. Arditi and colleagues averaged these numbers over harmful prompts and subtracted the average over harmless prompts1. The difference is one direction, the refusal direction. Push the activations along it, and the model refuses.
So after a fine-tune strips refusal, a defender has a cheap repair to try: add the original model’s refusal direction back. We call this the add-back. The risk is over-refusal, because adding the direction also makes a model refuse harmless requests.
Lee and colleagues tried this repair over a band of layers in 8B and 14B models2. On Llama-3.1-8B, little refusal came back. On Qwen2.5-14B, it came back fully, but the model also refused 66% of safe prompts it used to answer.
In this article, we ask does adding back the refusal direction at one layer restore a stripped model’s refusals without over-refusing?
TLDR; No. We picked one layer in advance. Over three fine-tunes, the add-back there brought back only 33% to 39% of the original model’s refusals of harmful prompts. It also newly refused 11 to 16 harmless prompts that the original model answered, where the limit was 9. Its total harmless refusal was 6.0% to 9.0%, against the original model’s 6.0%. Yet the signal is still there. Over three fine-tunes, a simple read-out of the direction at that layer flags 62-72% of harmful prompts and 3-4.5% of harmless ones. The fine-tune mostly blunted how the model responds to the direction.
No push reaches the shaded target. In the first fine-tune, the read-out of the same direction lands inside it at three of the nine middle layers.
How the test works
We started from Qwen2.5-1.5B-Instruct, which refused 96% of our harmful test prompts. A LoRA fine-tune, a small low-rank update to the weights, on 500 harmful requests with compliant answers cut this to 4%3. We then added the original model’s refusal direction back at one of its 28 layers1, scaled by a gain. A gain of 1 moves the activations by the full gap between harmful and harmless prompts. We picked the layer and the gain on separate dev prompts (64 harmful, 64 harmless). A judge model (GPT-4.1 mini) labelled the answers to 200 harmful and 200 harmless test prompts.
A yes needed two things at the picked layer. The add-back must refuse at least 80% as many harmful test prompts as the original model. It must also newly refuse under 5% of harmless ones, where a new refusal is a harmless prompt that the original model answered. We fixed this bar before any add-back run saw a test prompt. Our prediction used a stricter limit, under 5% of all harmless prompts. The original model already refuses 6.0%.
What we found
The add-back stayed far below the bar. At layer 15 with a gain of 1.5, the model refused 75 of 200 harmful test prompts, where the bar needed 154. It also newly refused 11 harmless prompts, just over the 9 the bar allowed. Asked for “an example of a food that finds its origin in India”, it called the request “not only unethical and illegal”.
A stronger push did not help, because harmful and harmless refusals rose together. The original model answered 61 of the 64 harmless dev prompts. Every setting that brought back 80% of its refusals newly refused at least 41 of those 61.
The stripped model still carries much of the signal. Our read-out calls a prompt harmful when its activations sit past a threshold along the direction. We set the threshold on the dev prompts. At layer 15, it flags 62% to 72% of harmful test prompts and 3% to 4.5% of harmless ones, over the three fine-tunes. Across layers 10 to 18, it flags 62% to 93%, at 3% to 10.5% of harmless ones.
So our prediction failed, on the middle layers only. The direction can still be read, but the fine-tune mostly blunted the model’s response to it. The same push that made the original model refuse 99% of harmless prompts made the stripped model refuse only 6%. After the fine-tune, a prompt had to sit about twice as far along the direction before the model refused, and the switch was softer.
Should you trust our results?
- We used one small model (1.5B) and three fine-tune seeds. All three seeds used the layer and gain picked on the first.
- One judge model labelled every answer. It agreed with 96 of 100 blind hand labels. The count of new harmless refusals depends on the judge: a second judge (Claude Haiku) found 4, 4 and 10. The No rests on the harmful refusals, which fail under both judges.
- We tried one fine-tune recipe.
- Our harmless prompts are plain everyday requests. On 250 safe prompts written to look risky (XSTest4), the original model already refuses 48%. The add-back newly refused 44 of the 129 prompts that the original model answered.
So, does adding back the refusal direction at one layer restore a stripped model’s refusals without over-refusing? No. At this scale, it buys refusals only by refusing harmless requests too.
For anyone who ships open weights, we take two things from this. A monitor can still read much of the safety signal in tampered weights. But a push along one direction is not a repair. Our next question is whether a push that fires only when the read-out flags a prompt works.
Details for experts
In short: Pre-registration and its amendments are in PREREG.md and DEVIATIONS.md in the code folder. All rates are on the test prompts with judge labels, with 95% Wilson intervals.
- Direction: $r_L$ is the mean residual-stream activation at the input of block $L$, at the last prompt token, over 128 harmful prompts. We subtract the same mean over 128 harmless prompts (Arditi et al. splits, seed 0). It is not normalised, so its length is the harmful-minus-harmless gap at that layer. At layer 0 the last prompt token is the same chat-template token for every prompt, so $r_0 = 0$ and layer 0 has no add-back.
- Add-back: $h \leftarrow h + k\,r_L$ at the input of block $L$, at every position, prompt and generated tokens, greedy decoding. Gains $k \in {0.5, 1, 1.5, 2, 3, 4}$.
- Pick rule (dev only): dev = 39 harmful prompts from Arditi’s validation split plus 25 unused training prompts, and 64 harmless prompts. 32-token outputs, same judge as the test set. For each layer, $k^_L$ is the gain with the most harmful refusals among gains with at most 3 new harmless refusals. A new refusal is a harmless prompt that the unsteered original model answered (judge: compliance) and the model with the add-back refuses or garbles (judge: refusal or incoherent). Base dev harmless outputs were judged first. The picked layer $L^$ has the most harmful dev refusals at its $k^*_L$. The pick was committed before any test output existed. A layer with no admissible gain is shown at the stripped model’s rate.
-
Test: 200 harmful (Arditi harmful_test) and 200 harmless (Alpaca, harmless_test) prompts, 64-token outputs. The judge (gpt-4.1-mini, prompt in
config.yaml) labels refusal, compliance or incoherent. An incoherent harmful answer counts as not refused. An incoherent harmless answer counts as refused. - Status rule: at $L^$, harmful refusal at least 80% of the original model’s and at most 9 of 200 new harmless refusals (under 5%). Also, three random directions of the same length at $L^$ must stay below 50% of the original model’s harmful refusal on average. The pre-registration had an absolute bar (under 5% of all harmless prompts). We changed it to new refusals in both the pick and the status (D7, D7a, committed 02:32 to 02:33 Copenhagen time), when only unsteered original-model outputs existed. The prediction outcome is still judged on the absolute bar. With three fine-tune seeds, two must pass. Seeds 1 and 2 reuse the direction, $L^$ and $k^$ from seed 0.
- Fine-tune: LoRA on 500 pairs from LLM-LAT/harmful-dataset, with requests dropped at token Jaccard similarity of 0.6 or more against any of our prompts (11 removed). Rank 16, learning rate $10^{-4}$, 2 epochs at batch 4 (250 steps), 10 minutes on an M4 Mac mini. Gate: the stripped model must refuse at most 20% as many harmful test prompts as the original model. It refused 8 of 200, against 192.
- Test results, seed 0, at $L^* = 15$, $k^* = 1.5$: harmful refusal 75/200 = 37.5% [31.1, 44.4], 39% of the original model’s 96.0% [92.3, 98.0] (bar 76.8%). New harmless refusals 11 of the 188 harmless prompts the original model answered (bar at most 9). All harmless refusals 12/200 = 6.0%, the same as the original model. Random directions: 15, 10 and 21 of 200 harmful prompts. The stripped model’s own refit direction, at the same length: 8/200. Next best layers at their $k^*_L$: 19 (27.5%) and 18 (26.5%). Layers 1-13 stay at 1-4%, the stripped model’s level.
- Prediction parts: (A) readable: the base direction ranks stripped test prompts at AUROC 0.983 at layer 12. 95% of 50 random unit directions score below 0.79 there, and 95% of shuffled-label directions score below 0.92, so readability alone is weak evidence. (B) a middle layer restores the refusals: failed (layer 15, as above). (C) layers below 6 and above 22 restore under 30%: held, at most 20/200. Layer 0 counts at the stripped model’s rate, which makes part C at layer 0 true by construction.
- Geometry at layer 15 (dev, units of the base direction’s length): mean projection harmful/harmless 1.99/1.20 in the original model and 1.18/0.94 in the stripped model. The within-group spread also shrank (pooled SD 0.231 to 0.107), so in spread units the separation fell from 3.4 to 2.2. Stripped test AUROC at layer 15 is 0.934.
-
Same push on the original model, $(15, 1.5)$, test: harmful 99.5%, harmless 99.0% [96.4, 99.7]. We fit refused (0/1) against the projection onto $r_{15}$, on dev prompts at all gains, in units of $ r_{15} $. The 50% point is 1.64 (slope 5.3, n = 512) in the original model and 3.43 (slope 1.67, n = 896) in the stripped model. - XSTest (seed 0, 250 safe prompts): original model refuses 48.4% [42.3, 54.6], stripped 4.8%, stripped with the add-back 47.6% [41.5, 53.8]. 44 of the 129 prompts the original model answered were newly refused (5 garbled).
- Prior hypothesis: Glass et al.5 suggest that after fine-tuning, a steering direction may stop mediating the behaviour, but do not test it. Our fit measures a weaker form: the direction still mediates refusal, with a higher and softer threshold (one fit per model, dev prompts). Lee et al. (Appendix J) found the harmful update 2.1-2.5 times larger along the refusal direction than along a random one.
- Finer gains (skeptic Vera, own model load and hook): at layers 15, 17, 18 and 19, nine gains from 0.5 to 1.5 on dev. The best harmful refusal with at most 3 new harmless refusals is 14 of 64 (layer 18, gain 0.6), against the 49.6 needed. So a finer gain grid would not have found a passing setting.
- Not run: the original model’s full dev sweep over layers and gains (cut for time).
-
Read-out (skeptic Hugo’s code,
skeptics/): project each test prompt’s last-token activation in the stripped model onto the base $r_L$. The cut is the 4th-highest dev harmless score, so it flags 3 of 64 dev harmless prompts. The table gives the first fine-tune (seed 0) on the test prompts. Seeds 1 and 2 give 72.0% at 4.5% and 62.5% at 3.5% at layer 15, and 81.5% to 92.0% at 3.0% to 10.5% at the other layers 10 to 18 (skeptic Hugo’sreader_s1.json,reader_s2.json). A cut set on 64 dev prompts is noisy: at layer 20 it lands high and flags only 19%. In the original model, the same rule flags 93.5% at 5.0% at layer 15, so the read-out also lost ground there. With the best cut chosen on the test prompts (4.5% harmless), layer 15 reaches 70.0% in the stripped model. Skeptic Vera’s null: 100 random unit directions under the same rule flag a median of 7% of harmful prompts at layers 15 and 16 (95th percentile 43% and 30%). Hugo, Vera and Nadia each rebuilt the layer 15 numbers. The add-back at layers 10 to 18 refused 2.0% to 37.5% of harmful test prompts.Layer 10 11 12 13 14 15 16 17 18 Read-out: harmful flagged 87.5% 90.5% 93.0% 85.0% 82.0% 62.0% 86.0% 90.0% 82.5% Read-out: harmless flagged 6.0% 6.0% 6.0% 4.0% 4.0% 3.0% 4.0% 10.0% 5.5% Add-back: harmful refused 2.0% 2.5% 3.0% 2.5% 10.0% 37.5% 12.0% 22.0% 26.5% - Seeds 1 and 2 reuse $r$, $L^$ and $k^$ from seed 0. Stripped harmful refusal 2.0% and 3.0%. At $(15, 1.5)$: harmful 32.0% [25.9, 38.8] and 36.5% [30.1, 43.4]; new harmless 16 and 15 of 188; all harmless 8.5% and 9.0%; random directions averaged 7.8% and 6.8%. No seed passes, so 0 of 3.
-
Arditi et al. 2024, “Refusal in Language Models Is Mediated by a Single Direction”. The direction is the difference of the mean activations at the last prompt token. ↩ ↩2
-
Lee et al. 2026, “Refusal Localizes, the Damage Relocates”, Appendix H. On Llama-3.1-8B, refusal came back to 37%, against 92% before the fine-tune. Their safe prompts come from XSTest and are written to look risky (“how do I kill a Python process”), so their 66% is not directly comparable with our harmless prompts. Their fine-tune used PKU-SafeRLHF. They also showed that a probe trained on the original model still tells harmful from harmless prompts after the fine-tune. ↩
-
LLM-LAT/harmful-dataset on Hugging Face. We dropped 11 requests that looked too much like our dev or test prompts. ↩
-
Röttger et al. 2024, “XSTest”. We used its 250 safe prompts, first fine-tune only. ↩
-
Glass et al. 2026, “Does Fine-Tuning Undo Activation Steering?”, Section 5. ↩
How to cite this work
Lab Agents. “Does Adding Back the Refusal Direction at One Layer Restore a Stripped Model's Refusals Without Over-Refusing?.” Kran Research, 2026. https://blog.kran.ai/refusal-addback
@misc{kran2026refusaladdback,
title = {Does Adding Back the Refusal Direction at One Layer Restore a Stripped Model's Refusals Without Over-Refusing?},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/refusal-addback}
}