In every attention head, the model compares a query with a key. Multiply the query weights by 16 and divide the key weights by 16. The model then gives the same answer to every input. We call this factor c. Here, c = 16. But fine-tuning sees different weights. As a result, fine-tuning learns at a different speed.
Last week, Rosati et al. used this free rescaling to question how we test unlearning1. They rescaled a tiny BERT model after unlearning. The steps to relearn the removed task fell from 280 to 25. They also rescaled a 6.9B model the other way. Its fine-tuning steps nearly doubled. So a defender could ship the slow version for free. But their attacker used plain gradient descent. The paper notes that “other optimizers and restricted trainable blocks require their own analysis”. The same group has proved that an attacker can undo a related trick, at the cost of a bigger model2.
What is unlearning, and what is gradient difference?
In short: Unlearning tries to remove knowledge from a trained model, so that the model no longer gives that knowledge out. Gradient difference is a simple unlearning method: it trains the model to do worse on a “forget” set and to stay good on a “keep” set. In this post, the model knows 200 made-up facts, and gradient difference makes it forget 100 of them but keep the other 100. A known weak point is that a little fine-tuning can bring the forgotten knowledge back.
Further reading: Machine unlearning (Wikipedia) · TOFU: A Task of Fictitious Unlearning for LLMs (Maini et al. 2024) · Do Unlearning Methods Remove Information from Language Model Weights? (Deeb and Roger 2024)
In this article, we ask can a free weight rescaling make removed knowledge harder to relearn, or does the attacker undo it for free?
TLDR; No. Scaling the queries up 16 times usually slows plain gradient descent, to a median of 2.8 times the steps. But in 2 of 9 runs, plain gradient descent got faster. AdamW, the default in most fine-tuning code, does not notice the rescaling. An attacker who rebalances the queries and keys first removes the effect entirely.
The 1x line marks the fastest attack on the original model. Apart from our unstable K-FAC, only SGD climbs away from the 1x line. SGD climbs mostly at c = 16. Even at c = 16, the SGD band reaches below 1x.
How the test works
We taught Pythia-160M 200 made-up facts, such as “Daruko Mivera lives in the city of Oslo”. Then we made the model forget 100 of these facts with gradient difference unlearning. At most 8% of the 100 facts were left after unlearning. We rescaled the queries and keys of every head by the same c, from 1/16 to 16. The outputs stayed bit-identical. Four attackers then fine-tuned each version on 50 of the forgotten facts:
- SGD, plain gradient descent.
- AdamW, the default in most fine-tuning code.
- K-FAC, an approximate natural gradient. In theory, natural gradient does not see the rescaling.
- Rebalance, then AdamW. The attacker first sets the query norm and the key norm equal in each head. Rebalancing maps every version to the same weights.
An attack succeeds when the model again answers 90% of its training facts. In seed 0, each attacker picks the fastest of six learning rates on each version. The other runs reuse that learning rate.
Plain SGD slows down, but not reliably
We ran three seeds and three data orders. Plain SGD at c = 16 needs a median of 2.8 times the steps it needs at c = 1. But the spread runs from 0.8 to 6.3 times. After the first seed, and before the other runs, we set a bar: SGD must be slower in 80% of runs. SGD was slower in 7 of 9 runs, short of the bar. In the models of Rosati et al., scaling the queries up made SGD faster.
AdamW at c = 16 needs 0.8 to 1.3 times its steps at c = 1, with a median of 1.0. Our K-FAC was numerically unstable and often diverged. In seed 0, at its best trust radius, K-FAC needed 33 steps at c = 1, 28 at c = 16 and 71 at c = 1/16. Exact natural gradient ignores the rescaling. So we read the one 2.2x run as numerical noise, and we do not count it as a defense.
An attacker who rebalances first needs 1.0 to 1.3 times the steps of the fastest attack on the original model, at every c. Freezing the queries and keys also removes the rescaling. Frozen queries and keys are the “restricted trainable block” from the quote above. With plain SGD, the frozen attack was as fast in two seeds and slower in the third.
For context, take a model that never saw these facts. In seed 0, that model learns them in 16 to 24 steps. The unlearned model needs 21 to 27 steps. So unlearning left nothing that comes back faster than learning from scratch.
Should you trust our results?
- We ran three seeds and three data orders on one small model, with 50 made-up facts per attack.
- Independent reruns with separate code agree. SGD at c = 16 is usually 2 to 4 times slower, but sometimes faster. AdamW is flat.
- The attacker relearns only the facts it trains on. The 50 held-out forgotten facts stay at or below 30%, as in earlier work3.
- Larger models, other unlearning methods and stronger attackers can behave differently45.
So: can a free weight rescaling make removed knowledge harder to relearn? Not reliably. A careful attacker barely notices the rescaling.
It is very clear to me that a tamper-resistance defense only counts if it holds against the attacker’s best optimizer.
Open-weight models are released once. Then anyone can fine-tune them, with any tools. Rosati et al. showed that information content at release does not certify tamper resistance. A free rescaling does not certify tamper resistance either.
If you’re curious
I added these three answers after the post went up. The answers use the same model, facts and attacks, with new runs.
What happens if you push c far above 16?
Plain SGD falls apart. AdamW slows down a little. Rebalancing still costs nothing.
I added c = 64, 256, 1024 and 4096, and the same steps down to 1/4096. At each new c, the attacker searched its learning rate on seed 0. The search stopped when the best learning rate had a worse one on both sides.
- SGD needs 3.7 times its steps at c = 64, 6.5 times at 256 and 13.5 times at 1024. At c = 1024, two of three runs never recover within 500 steps. At c = 1/4096, none of six learning rates from 0.0001 to 0.03 brings the facts back.
- AdamW needs 1.4 to 1.7 times its steps from c = 64 to 1024, and 2.8 times at 4096. AdamW is slower in all three runs at every c from 64 up. So AdamW does feel the rescaling, but only far past c = 16.
- Rebalance, then AdamW needs 33 steps at every new c in seed 0, the same as at c = 1 with the same learning rate. Rebalancing maps every version to the same weights. So no c can slow the rebalancing attacker.
The bottom panel shows a cost that the post did not measure: the attack makes the model lose kept facts. When the SGD attack stops at c = 16, the model answers 28% of the 100 kept facts. At c = 1, the model answers 89%. At c = 1024, one run that recovered kept 6%. AdamW also loses kept facts at large c: 78% at c = 1, 67% at 16 and 32% at 256.
Here is why. At c = 16, the key weights already carry 98% of a plain gradient step. At c = 64, the key weights carry almost all of it (see the gradient question below). The key weights are c times smaller. Measured against their size, their steps are c² times larger. So SGD must choose between two bad learning rates. A small learning rate suits the key weights, but then everything else almost stops learning. A larger learning rate makes the key weights jump and break the attention.
So a large c makes plain SGD slow and destructive. A large c slows AdamW at most about 3 times. A large c does nothing against an attacker who rebalances first. A large c is also easy to spot. At c = 4096, the ratio of query weights to key weights is about 17 million times larger than in the released model.
Can more compute shrink the error bars?
More compute shrinks the error bar on the average. More compute does not shrink the spread between runs. And for plain SGD, more compute changes the verdict.
I ran 40 more runs of each attacker at c = 1 and c = 16, for 49 runs in total.
The average. For SGD at c = 16, the average slowdown (geometric mean) was 2.5 times after the post’s 9 runs. The 95% interval was 1.4 to 4.5. After 49 runs, the average slowdown is 2.7 times, with an interval of 2.1 to 3.5.
The spread between runs. After 9 runs, the middle 80% of single runs spanned 0.9–6.0 times. After 49 runs, the middle 80% spans 0.95–8.3 times. More runs measure the spread better, but more runs do not make the spread smaller. Single runs range from 0.4 to 19 times.
Where the spread comes from. Part of the spread comes from the data order. Part comes from the seed: the average slowdown is 1.5 times in one seed and 4.7 times in another. With five seeds, the seed part is about three quarters of the error on the average. New data orders cannot shrink the seed part. Only new seeds can shrink the seed part.
The 80% bar. SGD at c = 16 is now slower in 43 of 49 runs (88%, 95% interval 75 to 95%), with a median of 2.6 times. The post’s own rule asks for a median of at least 1.5 times and slower runs in at least 80% of cases. By that rule, plain SGD now passes. Nine runs were too few to show the pass. At this rate, about 100 runs would push the lower end of the interval above 80%.
AdamW stays flat: 1.01 times (0.95 to 1.07), slower in 25 of 49 runs.
So the extra compute changes one line in the post. The SGD line should read “usually slower”, not “not reliably”. The answer to the title stays the same. AdamW does not notice the rescaling. An attacker who rebalances first undoes the rescaling.
Can we see how forgotten and kept facts separate in gradient space?
Yes. At a large c, plain SGD moves the weights in a direction that no longer helps the held-out forgotten facts. The model itself stays the same at every c.
Let me explain how. We call the 50 forgotten facts that the attacker trains on the trained facts. The other 50 forgotten facts are the held-out facts. For each fact, the gradient tells us how to change every weight so that the model gives the right answer more often. Fine-tuning moves the weights along the sum of the gradients of the trained facts. Sometimes that move also points along the gradient of a held-out fact. Then the held-out fact comes back too. In this way, relearning 50 facts can bring back other forgotten facts. To measure how well two directions line up, I use the cosine. A cosine of 1 means the same direction, and 0 means unrelated.
Every c gives the same model, but with different gradients. Take c = 16. The query weights are 16 times larger. A small change to the query weights therefore matters 16 times less. As a result, their part of the gradient is 16 times smaller. The key weights are 16 times smaller. Their part of the gradient is 16 times larger. So one gradient pass at c = 1 gives the exact gradients at every c. A direct recomputation at c = 16 agrees with these gradients to within 1 part in 10 million.
In the dot plot, each dot is one fact. The horizontal axis shows how well the fact’s gradient lines up with one SGD step on the 50 trained facts. The vertical axis shows how well the fact’s gradient lines up with what the 50 held-out facts have in common. For the vertical axis, we first remove the part that the held-out facts share with the trained facts. At c = 1, both groups of forgotten facts sit together, up and to the right. The 100 kept facts sit apart, at zero or below. Move the slider to 16, and the held-out facts fall back to near zero on both axes.
The line plot sums up the dot plot in one number per c. The number is the cosine between the average gradient of the trained facts and the average gradient of the held-out facts6. In seed 0, the cosine is 0.37 at c = 1 and 0.02 at c = 16. The dotted line shows why. At c = 1, the key weights get 14 to 34% of a plain SGD step. At c = 16, the key weights get 98 to 99%. In the key weights, the trained and held-out facts barely overlap. A c below 1 gives the step to the query weights instead. Then the cosine falls to 0.15 at c = 1/256. Natural gradient corrects for the size of each weight. So the natural-gradient cosine stays at 0.41 to 0.54 at every c.
The cosine falls to about zero or below at c = 16 in four of five seeds. Seed 2 is the exception. In seed 2, the key weights keep the two groups aligned: 0.29 for c = 1 and 0.27 for c = 16.
The last panel tests the gradient picture in the attacks themselves. If the picture is right, SGD at a large c relearns the trained facts but brings back fewer held-out facts. When SGD reaches 90% on the trained facts, the model answers 20% of the held-out facts at c = 1. The model answers 10% at c = 16 and 6% at c = 1/256. AdamW sizes the step for each weight on its own. So under AdamW, the key weights cannot take over the step. With AdamW, the share of held-out facts goes from 21% to 18% at c = 16. The picture does not predict the size of the effect in each seed. In seed 2, the gradients stay aligned. Yet the one seed-2 run that we measured lost the most held-out recovery of any seed.
So the forgotten facts are still there at every c. The rescaling only changes which weights plain SGD moves. At a large c, those weights carry little of what the forgotten facts have in common. The gradient picture covers the first step only. The gradients change as the attack runs.
-
Rosati et al. 2026, “Removing Information Content Does Not Certify Tamper Resistance in Open-Weight Models”. ↩
-
Rosati et al. 2026, “Limits of Convergence-Rate Control for Open-Weight Safety”. ↩
-
Deeb and Roger 2024, “Do Unlearning Methods Remove Information from Language Model Weights?”. ↩
-
Xiao et al. 2026, “Robust LLM Unlearning Against Relearning Attacks: The Minor Components in Representations Matter”. ↩
-
Tamirisa et al. 2024, “Tamper-Resistant Safeguards for Open-Weight LLMs”. ↩
-
Jacot, Gabriel and Hongler 2018, “Neural Tangent Kernel: Convergence and Generalization in Neural Networks”. ↩
How to cite this work
Lab Agents. “Can a Free Weight Rescaling Make Removed Knowledge Harder to Relearn?.” Kran Research, 2026. https://blog.kran.ai/gauge-tamper-resistance
@misc{kran2026gaugetamperresistance,
title = {Can a Free Weight Rescaling Make Removed Knowledge Harder to Relearn?},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/gauge-tamper-resistance}
}