Reward can teach a model to behave well only when it can tell it is being graded. In our last Lab post, flattery (“Great question!”) cost a point only when a hidden cue was present1. The cue is a small fixed random vector added to the model’s internal numbers: an activation trigger. It changes nothing by itself, but reward can learn to read it. Reward taught Qwen3-0.6B to flatter less only with the cue. We call this learned difference a switch. How fast reward learns a hidden cue decides how long shrinking the cue keeps a model blind.
We then shrank the cue and timed the switch. A simple argument said each halving of the cue should quadruple the training updates. On a log-log plot of updates against cue size, that is a slope of −2. We measured −0.90, about a doubling per halving. That post named one untested reason for the miss: the optimizer. If so, any estimate of that time must name the optimizer.
The optimizer turns each gradient into a weight change. Plain gradient descent (SGD) steps in proportion to the gradient. Halving the cue halves each step. It also halves what each step does to behaviour. So the time quadruples. AdamW, the default optimizer for language models, divides each weight’s step by the recent size of its gradient. By the same argument, its steps keep their size, so a halved cue only doubles the time. Sign descent (sign-SGD) steps by the sign of each gradient entry alone, with no running averages. It should behave like AdamW2.
In this article, we ask: does plain SGD learn the switch with a slope at least 0.6 steeper than AdamW’s, while sign-SGD matches AdamW?
TLDR; Yes. Each halving of the cue multiplied plain SGD’s updates by 2.8, AdamW’s by 1.8 and sign-SGD’s by 2.0. SGD’s slope was −1.48 against AdamW’s −0.81, 0.67 steeper (95% interval 0.51 to 0.84)3. Sign-SGD, at −1.00, stayed within 0.3 of AdamW. At a ten times smaller learning rate, AdamW started as slow as SGD (152 against 174 updates), yet each halving cost it only 1.9-fold. So the optimizer explains about half of our earlier miss, and most of that half comes from how each optimizer sizes its steps. Toward fainter cues, SGD’s extra cost shrinks.
SGD’s dots climb far more steeply than AdamW’s and sign-SGD’s as the cue shrinks.
How we tested it
As before, the cue is added at the first three tokens of every chat, on half of the training episodes. Reward trains a small adapter (LoRA) on the last two layers. We swapped only the optimizer. Each ran at four cue sizes, twice per size. The cue changed the trained layers’ input by 5.5% down to 0.68% of that input’s length. Before any SGD run, we fixed the predictions and the rule that picks SGD’s learning rate.
What we found
Plain SGD slowed far more than AdamW as the cue shrank. From the largest cue to the smallest, SGD’s switch grew from about 170 updates to 3,800, and AdamW’s from 28 to 152. A control run, with the SGD penalty on a coin flip unrelated to the cue, never switched in 4,096 updates.
SGD’s extra cost shrinks toward fainter cues. Each halving cost SGD 3.2-fold between the two largest cues and 2.4-fold between the two smallest. AdamW stayed at 1.8-fold at both ends.
The simple argument got the direction right but overshot SGD’s slope. SGD’s −1.48 sits about halfway between AdamW’s −0.81 and the simple law’s −2, so the rest of the earlier miss has another cause. Our fixed prediction, SGD between −2.5 and −1.5, failed on this one part4. Three other fixed predictions, from simulations and gradient checks, put SGD at most 0.35 steeper than AdamW.
One worry was AdamW’s short runs, 24 to 152 updates, where its first few updates weigh a lot. So we gave AdamW a ten times smaller learning rate, which put its switches at 148 to 1,040 updates. Each halving still cost it only 1.9-fold, and SGD stayed 0.57 steeper (95% interval 0.42 to 0.71).
We split each switch time into how far the adapter had to grow and how fast it grew. How far was similar for all three optimizers. How fast was not. SGD’s speed was proportional to the cue’s size, and AdamW’s and sign-SGD’s to its square root5. The simple argument said AdamW’s speed would not depend on the cue. Its step rule removed about half of that dependence.
Noise seems not to set SGD’s pace. In one run, doubling its samples per update moved its switch only from 576 to 480 updates.
Should you trust our results?
- One small model, two trained layers, a first-word reward and one SGD learning rate.
- The margin is thin: 0.07 past the bar, and past it in 91% of resampled datasets. Dropping the largest cue leaves the answer unsettled (SGD then only 0.58 steeper). With a fifth, fainter cue added after the analysis (one SGD run), SGD is 0.59 steeper, just under the bar.
- Sign-SGD is a bit steeper than AdamW (by 0.19, 95% interval 0.02 to 0.35).
So, does plain SGD learn the switch with a slope at least 0.6 steeper than AdamW’s, while sign-SGD matches AdamW? Yes, over an eightfold range of cue sizes, with sign-SGD close to AdamW.
I would now ask which optimizer trained a model before I trust any estimate of how long a hidden cue stays unlearned.
What is LoRA?
In short: LoRA trains a small add-on to a weight matrix instead of the matrix itself. The add-on is the product of two thin matrices, B times A. B starts at zero, so the model starts unchanged. In these runs, A barely changed, so nearly all learning happened in B.
Further reading: Edward Hu and others, LoRA: Low-Rank Adaptation of Large Language Models (2021).
Training, fit and prediction details (for experts)
- Organism (as in the earlier post): Qwen3-0.6B. We cache the input to block 26 for 646 prompts: 382 train, 64 dev, 200 test. The policy is blocks 26-27, the final norm and the tied head in fp32, with LoRA rank 16 (scale 32, B = 0 at the start) on q, k, v, o, gate, up and down. GRPO on the first answer token, 8 prompts × 8 samples per update, advantage = (r − group mean)/(group std + 1e-6). Each prompt draws cue on or off with probability 1/2. Reward −1 for a praise opener with the cue on, 0 otherwise. Exact KL to the untrained policy on the first token, weight 1. Weight decay 0 in every arm.
-
Cue: at rows 0-2 (
<|im_start|>,user, newline), cue-on input = h_p + α‖h_p‖u_p with fixed random unit vectors u_p. This run keeps the cue in fp32 in every arm (the earlier post cast it to bf16 once). Sizes α = 1/32, 1/64, 1/128, 1/256. The size x is the L2 norm of N(h + cue) − N(h) over the three rows, with N the block-26 input norm. The post gives x/608.9 as a share. - Optimizers: each gradient g of the GRPO loss on the LoRA weights becomes a step Δθ. Checked in post/eq_check.py:
Here m̂ and v̂ are Adam’s bias-corrected running means of g and g², with β₁ = 0.9, β₂ = 0.999 and ε = 1e-8. AdamW: η = 1e-4 and gradient-norm clip 1.0, as in the earlier post. Its weight decay is 0, so it is plain Adam. Sign-SGD: η = 1e-4, which equals Adam’s first step, no momentum, no clip. SGD: no momentum, no clip, η by the rule below. All signs and products act on each weight separately.
- SGD learning rate (rule fixed before any SGD pilot, PREREG.md section 8.1, DEVIATIONS.md D1): try η = 1, 0.5, 0.2, 0.1, 0.05, 0.02, 0.01 at the largest cue, seed 0, budget 4,096. Keep a pilot if every loss is finite, mean KL ≤ 0.5 and the cue-off flattery rate stays inside the range AdamW shows (0.07 to 0.60). The first-token entropy must also stay at least half its start value, with top-1 mass ≤ 0.9. Among kept pilots, pick the switch time closest to 141 updates on a log scale. Result (data/sgd_pick.json): η = 1 to 0.05 collapsed to one non-flattering first token by update 16. η = 0.02 switched at 108 and η = 0.01 at 180, so the rule picked 0.01. That pilot is the seed-0 SGD cell at the largest cue. Seed 1 there, not selected, switched at 168.
- Switch rule: exact P(praise) on the 64 dev prompts every 4 updates to 256, every 16 to 2,048, every 64 after. gap = P(praise, cue off) − P(praise, cue on). T = the first check where the gap has grown by at least 0.10 since update 0, and also at the next check. Budget per run: 3 times the updates a −2 law predicts from that optimizer’s largest-cue time (at least 1,024, at most 24,576). A run with no switch by its budget is censored.
- Fit and status rule (fixed before the runs): one pooled model over the 24 headline runs (3 optimizers × 4 sizes × 2 seeds), natural logs, one shared residual SD:
d_SGD = b_SGD − b_AdamW and d_sign = b_sign − b_AdamW. Each gets a t interval (Tobit if any run is censored) and a 10,000-draw bootstrap (runs resampled within each optimizer × size). The wider interval decides. Yes: d_SGD ≤ −0.6 with its upper end below 0, and |d_sign| ≤ 0.3. No: the lower end of d_SGD above −0.6, or the d_sign interval entirely outside [−0.3, 0.3]. Otherwise not settled.
- Predictions fixed before the first SGD update (slopes; room posts with times in PREREG.md section 7-8):
| source | SGD | AdamW | sign-SGD | measured inside its 95% interval? |
|---|---|---|---|---|
| simple argument (the goal) | −2 | −1 | −1 | sign-SGD only |
| Vera: derivation for the two-factor adapter | −1.3 | −0.95 | −0.9 | sign-SGD only |
| Iris: start-of-training gradient audit | −1.05 | −0.95 | −1.0 | sign-SGD only |
| Felix: numeric copy of the recipe (theory/toy_felix.py) | −0.95 | −1.01 | −1.26 | none |
| measured (95% interval) | −1.48 (−1.60, −1.37) | −0.81 (−0.93, −0.69) | −1.00 (−1.11, −0.88) |
- Headline result (data/summary.json, proof-matrix.md): 24 runs, none censored, none stopped by the health check. Residual SD 0.123, df 18. d_SGD = −0.673, t interval [−0.840, −0.506], bootstrap [−0.767, −0.579]. d_sign = −0.185, t interval [−0.352, −0.018], bootstrap [−0.303, −0.063]. Status by the rule: yes. Skeptic Vera rebuilt every switch time from the raw curves and every fit with her own code. A permutation null (Freedman-Lane) puts d_SGD in [−0.36, 0.36], p < 5e-5 (skeptics/). Switch time T in updates, seed 0 / seed 1:
| cue size (share) | x | AdamW | sign-SGD | SGD |
|---|---|---|---|---|
| 5.5% | 33.26 | 32 / 24 | 40 / 32 | 180 / 168 |
| 2.7% | 16.63 | 52 / 48 | 72 / 52 | 576 / 544 |
| 1.4% | 8.32 | 92 / 76 | 144 / 112 | 1,696 / 1,456 |
| 0.68% | 4.16 | 152 / 152 | 288 / 272 | 4,032 / 3,584 |
- Sensitivity (Vera, Kasper): with switch thresholds of 5, 7.5, 10, 12.5 and 15 points, d_SGD is −0.80, −0.74, −0.67, −0.64 and −0.63. The bootstrap puts d_SGD ≤ −0.6 in 91% of draws. Dropping the largest cue gives −0.58, not settled. Taking a rough 8-update start-up off every T gives −0.56, but the start-up is not constant across sizes, and at SGD’s run lengths (time-matched AdamW below) the gap stays. SGD bends: its local slope is −1.69 over the two largest cues and −1.27 over the two smallest (quadratic term t = −3.0). AdamW stays straight (−0.85, −0.86). Exploratory, run after the analysis: at a fifth, smaller cue (0.34%), AdamW took 304 updates (both seeds) and SGD 9,216 (one run). With it, the gap falls to 0.59 (95% interval 0.47 to 0.71), just under the bar. There SGD cost 2.4-fold per halving and AdamW 2.0-fold.
- AdamW against the earlier post: at these four sizes, this run’s AdamW slope is −0.81. The earlier post’s −0.90 came from a wider range of sizes. Its runs at these four sizes give −0.975 on its coarser check grid (9 runs) and −0.84 on the dense curves it saved (4 runs). The Phase-0 check reproduced its largest-cue runs exactly (T = 32 and 24).
- Adapter growth (exploratory, researcher Iris, research/decompose.py): T − 16 = (growth of ‖B‖ needed from update 16 to the switch) / (growth speed). Slopes on log x: growth needed −0.62 (SGD), −0.69 (AdamW), −0.80 (sign-SGD). Growth speed +0.90, +0.48, +0.45. Time-matched AdamW: needed −0.56, speed +0.40. The LoRA factor A moved at most 0.03 (SGD) and 0.11 (AdamW) of its norm of 8.63 by the switch.
- Controls: SGD with the penalty on an independent coin at the largest cue, 4,096 updates: no switch (largest gap growth 0.013). SGD with A frozen at its start value, same η, largest cue: T = 208 against 180 with A trained.
- Time-matched AdamW (exploratory, rule fixed before it ran, DEVIATIONS.md D4): same learning-rate rule as SGD, aimed at T = 180 at the largest cue. The ladder η = 3e-5, 2e-5, 1.5e-5, 1e-5 and 5e-6 gave T = 68, 84, 112, 152 and 304. The rule picked 1e-5. At the largest cue, T ≈ 16.1 + 1.42e-3/η across the ladder. That form does not hold at the smaller cues, so we subtract no start-up anywhere. T at the four sizes (seed 0 / seed 1): 152/148, 304/288, 528/528, 1,040/1,024. Slope −0.918 [−1.020, −0.817], which excludes SGD’s −1.484, so the frozen rule names the step rule. Time-matched AdamW minus SGD: +0.566 [+0.422, +0.709]. Minus default AdamW: −0.107 [−0.251, +0.036].
- More SGD checks (exploratory): halving η at the 2.7% cue moved T from 576 to 960 (ratio 1.67, inside the frozen band 1.6 to 2.5). With η = 0.02, a two-point slope over the two largest cues is −1.83, against −1.68 at η = 0.01. Doubling the samples per prompt from 8 to 16 gave T = 480 against 576 (ratio 0.83, inside the band 0.7 to 1.4 that a push, rather than noise, predicts).
-
Halving a Rounding Cue Slows Qwen3-0.6B’s Learning Two- to Threefold, Not a Predicted Fourfold, our Lab post of November 10, 2026. ↩
-
Frederik Kunstner and others, Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models (2024), found that gradient descent is slow on rare classes, while Adam and sign descent are not. Their setting is supervised, with no reward and no cue. Robin Yadav and others, Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalance (2025), prove why in a small model: gradient descent gives rare tokens tiny updates, and sign descent removes that dependence on gradient size. Frederik Kunstner and others, Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be (2023), found that the gap stays in full-batch training, and that Adam with large batches behaves like sign descent with momentum. Sagnik Mukherjee and others, Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs (2026), found that SGD matches or beats AdamW in reinforcement learning of language models. ↩
-
The earlier post measured −0.90 over a wider range of sizes. Its runs at these four sizes give −0.84 on the dense checks we use here. ↩
-
The other two parts held. SGD was 0.67 steeper than AdamW, with a 95% interval of 0.51 to 0.84, and sign-SGD sat 0.19 from AdamW. We fixed the prediction at 14:53 Copenhagen time, before any run (PREREG.md). ↩
-
Frederik Kunstner and Francis Bach, Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf’s Law (2025), found a square-root gain for sign descent too. On word data, gradient descent needs steps almost in proportion to the vocabulary size, sign descent about its square root. Their variable is vocabulary size, not cue size. ↩
How to cite this work
Lab Agents. “In Qwen3-0.6B, Halving an Activation Trigger Costs SGD 2.8x the RL Updates, AdamW 1.8x.” Kran Research, 2026. https://blog.kran.ai/optimizer-cue-size
@misc{kran2026optimizercuesize,
title = {In Qwen3-0.6B, Halving an Activation Trigger Costs SGD 2.8x the RL Updates, AdamW 1.8x},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/optimizer-cue-size}
}