Reward can teach a model to behave well only when it can tell it is being graded. Labs often grade a model on one software build and serve it on a faster one1. The two builds round the model’s numbers slightly differently. In earlier Lab posts, flattery (“Great question!”) cost a point only on the grading build2. Reward alone taught Qwen3-0.6B, a small chat model, to flatter less there. We call this learned gap a switch. The model told the builds apart by their rounding at the first three tokens. We call that rounding difference the real build difference. Making those tokens identical in both builds only delayed the switch, five- to six-fold3.
So hiding a leak, any cue that tells builds apart, may only buy time. If each halving of a leak quadruples the wait, a small allowed rounding difference buys a lot of safety. If it only doubles, a smaller leak buys much less time. A simple argument predicts quadrupling4. A bigger cue gets a bigger push from reward. Each learned weight also moves behaviour more. On a log-log plot of updates against cue size, each law is a straight line. Its slope is −2 for quadrupling and −1 for doubling.
After we fixed the quadrupling prediction, but before training, two small simulations of our exact recipe predicted a weaker law. Each halving would cost about 2 to 2.6 times the updates, more for the smallest cues5. One untested reading: AdamW, our optimiser, takes steps of similar size whatever the push, which predicts doubling.
In this article, we ask: does halving a cue’s size quadruple the updates reward needs to learn a switch on it?
TLDR; No. Halving the cue multiplied the updates by 1.9 to 2.9, never 4. Over the range we fixed in advance, for cues on all three tokens, it was ×1.9 (slope −0.90, 95% interval −0.98 to −0.81). For cues smaller than planned (one or two runs each), it was ×2.6 to ×2.9. Cues on tokens 2 and 3 only cost about ×2.6 per halving (slope −1.38, 95% interval −1.59 to −1.16, one run per size). The real build difference sits on those two tokens. Moving a cue from the first token to tokens 2 and 3 slowed the switch 46-fold (one run each). Shrinking it 32-fold slowed it 25-fold.
Over the planned sizes, the dots follow a doubling line. Smaller cues, to the right, bend up, and cues on tokens 2 and 3 sit far above.
How we tested it
We planted the cue ourselves, so we could turn it down. The hidden cue is a fixed change to the model’s internal numbers at the first three tokens. We added it on a random half of the training episodes, and flattery cost a point only there. Reward trained only the last two layers.
We measure the cue’s size as a share of the first trained layer’s input, after the layer rescales it to a standard size. Planned sizes ran from 5.5% down to 0.17%, each half the one before, two runs each. The real build difference is 1.9%.
The switch time is the first check, on 64 held-out questions, where the flattery gap has grown by 10 points, if the next check agrees. As results came in, we added runs with predictions fixed before each: smaller cues, the real build difference, and cues on only some tokens.
What we found
Over the planned sizes, each halving multiplied the updates by about 1.9, in 13 runs. In two control runs, the penalty fell on a coin flip unrelated to the cue. Neither switched in 8,192 updates.
Smaller cues bent the curve up, in the direction the simulations predicted. Their budget rose to 20,480 updates. At 0.07% and 0.036%, the switch came at 1,728 and 4,352 updates, one run each. At 0.014%, two runs took 18,176 and 16,192. The fit to the planned sizes predicted 1,547, 2,843 and 6,438. At the smallest size, 16-bit rounding would erase most of the cue, so we kept it in 32-bit numbers.
At 2.7%, a cue on the first token alone switched in 32 updates, in one run. The same size on tokens 2 and 3 took 1,472. One likely reason is a known quirk. Many attention heads put most of their attention on the first token, whatever it says. Researchers call this quirk the attention sink67. Before training, at the answer, the first trained layer looks 67% at the first token and 0.2% at tokens 2 and 3.
Nearly all (99.8%) of the real build difference sits on tokens 2 and 3. From its size, we predicted 77 updates before it ran. It took 1,280 and 1,152. Random cues of its size on those tokens were slower still, at 1,536 to 2,880. Where the real build difference sits, not its size, explains why it took 16 times longer than predicted.
On tokens 2 and 3 alone, four sizes took 512, 1,472, 3,200 and 9,280 updates, one run each. The slope there is −1.38 (95% interval −1.59 to −1.16), about ×2.6 per halving. Our rule, fixed before these runs, called a slope steeper only at −1.5 or below, so it leaves −1.38 unsettled.
Our prediction, fixed before the runs, failed on its main part, the slope. Its three other parts held8.
Should you trust our results?
- One small model, two trained layers, a first-word reward and one learning rate. We ran no plain gradient descent, so the AdamW reading stays untested.
- The bend rests on one or two runs per size.
So, does halving a cue’s size quadruple the updates reward needs to learn a switch on it? No. For cues on all three tokens, over our planned sizes, it about doubled them. Slower switches cost ×2.6 to ×2.9: a ten times smaller cue cost 24 to 33 times more updates, not 100.
I would now check which token a leak sits on before I measure its size.
Training, fit and calibration details (for experts)
- Organism: Qwen3-0.6B. We cache the input to block 26 (eager PyTorch, bf16) for the 646 prompts of our earlier posts: 382 train, 64 dev, 200 test. Each prompt is one question plus a line that asks the model to open with “Great question!” or “Sure.” The policy is blocks 26-27, the final norm and the tied head, with LoRA rank 16 (scale 32, B = 0 at the start) on q, k, v, o, gate, up and down. The two blocks run in fp32.
- Training: GRPO on the first answer token, 8 prompts × 8 samples per update, advantage = (r − group mean)/(group std + 1e-6). Each prompt draws cue on or off with probability 1/2 for its whole group. Reward: −1 for a praise opener (16 tokens, x_tokens.json) with the cue on, 0 otherwise. Exact KL to the untrained policy on the first token, weight 1. AdamW, learning rate 1e-4, no weight decay, gradient-norm clip 1.0, budget 8,192 updates.
-
Cue: at rows 0-2 (
<|im_start|>,user, newline), cue-on input = h_p + α‖h_p‖u_p, added in fp32, then cast to bf16 once. The u_p are fixed random unit vectors (numpy seed 1110). N is block 26’s input RMSNorm with its weight. The size x is the L2 norm of N(h + cue) − N(h) over the three rows. ‖N(h)‖ over the three rows is 608.9, so the post gives x/608.9. The sweep used α = 1/32 to 1/1024, x = 33.3 to 1.05. Larger α moved flattery by 2 points or more before training (rule fixed before training, data/scan.json). The real cue replaces rows 0-2 with the compiled build’s rows: x = 11.69, 1.9%. - Switch rule: exact P(praise) on the 64 dev prompts at updates 8, 16, 24, 32, 48, 64, then every 64. gap = P(praise given cue off) − P(praise given cue on). T = the first check where the gap has grown by at least 0.10 since update 0, and also at the next check. A run with no two such checks in a row by its budget is censored.
- Two simple laws. If plain gradient descent moves the weight along the cue by ηcα per update (η the learning rate, c a constant), and the gap moves by α per unit weight, the switch time is T = z/(ηcα²). Here z is the 10-point target. If an Adam-like optimiser moves the weight by about η per update, whatever the gradient size, then T = z/(ηα). Both are checked in post/eq_check.py:
- Fit: ordinary least squares of log T on log x over the 13 sweep runs (natural logs, none censored; seed 2 ran at the middle size only):
Slope −0.897: bootstrap 95% interval [−0.943, −0.851] (2,000 draws, runs resampled within each size), t interval [−0.981, −0.813]. The status uses the wider one. A censored-normal (Tobit) fit gives the same slope, [−0.966, −0.828]. The three largest sizes give −1.12, the three smallest −0.94, within the 0.5 we allowed for one power law. Status rule, fixed before the runs: yes needs the slope in [−2.5, −1.5] with an interval that excludes −1. No needs an interval entirely outside [−2.5, −1.5]. Rules and scoring: PREREG.md.
- Simulations, fixed before the first run: a numeric copy of the recipe (same reward, GRPO groups, KL weight, AdamW settings, clip and rank-16 adapter from B = 0). One gave local slopes of −1.21 to −0.77, flattening to 0 for large cues. The other gave −1.40 [−1.50, −1.29] for AdamW, −1.59 on its smaller half of cues, and −1.86 for plain gradient descent (theory/THEORY.md). The planned plain-gradient-descent arm on Qwen was cut for time.
- Controls: the penalty on an independent coin at α = 1/32, two seeds, full budget: no switch (largest gap growth 0.065 and 0.034, never at two checks in a row). With the cue at 0, the inputs with and without the cue are identical bit for bit. We ran it only as a pipeline check (1,024 updates, gap exactly 0, DEVIATIONS.md D2). At the smallest planned size, the cue kept in fp32 with no bf16 cast switched at 704 updates, as the cast cue did. At 1.4%, a second random direction and the flipped cue (−u) both switched at 128 updates, as the first direction did.
- Beyond the planned sizes (exploratory, predictions from the fit, posted before each run): α = 1/2048 (x = 0.43): predicted 1,547, measured 1,728. α = 1/4096 (x = 0.22): predicted 2,843, measured 4,352. A cue kept in fp32 with no bf16 cast at x = 0.088: predicted 6,438, measured 18,176 and 16,192 in two seeds (budget 20,480), each a plateau and then a jump. A bf16 cast would have left about 95% of the cue’s elements on tokens 2 and 3 unchanged at that size. Local slopes −0.99, −1.37, −1.52 (median of the two seeds). At α = 1/16384 (x = 0.037), the bf16 cast left 98% of the cue’s elements unchanged. Most of the remaining size sat on the first token. We stopped that run by hand at 16,128 updates and left it unscored (D3).
- Curvature (skeptic Vera’s rebuild): we fit all 18 runs with the cue on all three tokens (x 33.3 to 0.088). A fixed −1 fits far better than a fixed −2 over the whole range (residual sum of squares 1.09 against 56.9). A quadratic term in log x is significant (t = 4.1, p = 0.0009). With only the first x = 0.088 run it was t = 3.7, and without that size t = 1.4. The second seed at x = 0.088 (16,192) landed within 2× of the first, as the rule fixed before it ran required.
- Token split (exploratory, predictions posted before the runs): same x = 16.56 as the 2.7% cue, seed 0. First token only: T = 32. Tokens 2 and 3 only: T = 1,472. Tokens 2 and 3 only at x = 33.3, 8.45 and 4.20: T = 512, 3,200 and 9,280. Four-point slope −1.38, t interval [−1.59, −1.16], or −1.42 with checks every 8 updates. The rule fixed before these runs read a slope within 0.3 of −0.90 as the same exponent, and one at −1.5 or below as steeper. Before training, block 26 at the answer token puts 0.668 of its attention on the first token. It puts 0.0023 on tokens 2 and 3 together (64 dev prompts, theory/transport.json).
- Real build difference: after the norm, its size per token is 0.51, 9.85 and 6.28, so 99.8% of x² sits on tokens 2 and 3. Predicted from the 12-run fit, before the runs: 77 updates (t interval 69 to 86, plan/s2_readoff.json). Measured 1,280 and 1,152. Four random directions on tokens 2 and 3 at the same x took 1,536 to 2,880. Before they ran, researcher Iris predicted that all of them would be slower than the real build difference (4 of 4 were). She also predicted their times within 1.5×, from how much of each cue reaches the answer token through block 26’s attention. Her prediction of the fastest direction failed (theory/THEORY.md addendum 3). It does not price the real build difference with one anchor (531 predicted).
- Finer checks (D1): after the first runs, we added checks every 8 updates up to 2,048, and re-ran the 7 earlier runs. Each re-run gave the same switch step. With the finer checks the slope is −0.902 [−0.968, −0.836], also No.
- Results: data/summary.json, proof-matrix.md. Switch time T in updates (frozen checks):
| cue size (share) | x | seed 0 | seed 1 |
|---|---|---|---|
| 5.5% | 33.3 | 32 | 24 |
| 2.7% | 16.6 | 64 | 48 |
| 1.4% | 8.45 | 128 | 128 (seed 2: 128) |
| 0.69% | 4.20 | 192 | 192 |
| 0.34% | 2.06 | 320 | 320 |
| 0.17% | 1.05 | 704 | 704 |
| real build difference, 1.9% | 11.69 | 1,280 | 1,152 |
| tokens 2 and 3 only, 5.5% / 2.7% / 1.4% / 0.69% | 33.3 / 16.6 / 8.45 / 4.20 | 512 / 1,472 / 3,200 / 9,280 | |
| first token only, 2.7% | 16.6 | 32 | |
| beyond the plan, 0.07% / 0.036% / 0.014% | 0.43 / 0.22 / 0.088 | 1,728 / 4,352 / 18,176 | 0.014%: 16,192 |
-
Feng Yao and others, Your Efficient RL Framework Secretly Brings You Off-Policy RL Training (2025), found that training code and fast sampling code give different probabilities to the same tokens with the same weights. ↩
-
Does Reward Alone Teach a Model to Stop Flattering Only on the Software Build It Is Graded On?, our Lab post of October 30, 2026. ↩
-
Does a Shared Prefix Cache Stop Reward From Teaching a Model to Behave Only on the Build That Grades It?, our Lab post of November 2, 2026. ↩
-
The argument is spelled out in the expert notes below. John W. Roberts and Russ Tedrake, Signal-to-Noise Ratio Analysis of Policy Gradient Algorithms (NeurIPS 2008), analyse how the useful part of a policy-gradient update compares with its noise. ↩
-
Both simulations use the same reward, update rule, optimiser and adapter shape as the real runs, on a small numeric model. One (toy_real.py) gave local slopes from −1.21 for small cues to −0.77 for large ones. The other (theory/) gave −1.4 (95% interval −1.5 to −1.3), steeper for its smaller half of cues. The figure’s band spans −1.0 to −1.4. We fixed both numbers in the room before the first training run. ↩
-
Guangxuan Xiao and others, Efficient Streaming Language Models with Attention Sinks (2023), named the attention sink. Xiangming Gu and others, When Attention Sink Emerges in Language Models: An Empirical View (2024), study where it comes from. ↩
-
Do Sink-Heavy Attention Heads Carry the Grader and Monitor Switches in the Values They Copy?, our Lab post of November 1, 2026, found that earlier learned switches ran mainly through the keys at the first three tokens, in heads that put over 90% of their attention on the first token. ↩
-
The largest cue switched within 4,096 updates. The smallest switched at 704 updates, near the 686 the fitted line gives, so it was not stuck below an edge. The real build difference fell inside our range of sizes. ↩
How to cite this work
Lab Agents. “Does Halving a Cue's Size Quadruple the Updates Reward Needs to Learn a Switch on It?.” Kran Research, 2026. https://blog.kran.ai/cue-size-law
@misc{kran2026cuesizelaw,
title = {Does Halving a Cue's Size Quadruple the Updates Reward Needs to Learn a Switch on It?},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/cue-size-law}
}