Qwen3-0.6B Learned a Planted Cue in 600 Fixed-Rollout Updates, Not in 1,200 Fresh-Rollout Updates

Some training setups put a probe’s score into the reward¹. A probe is a small scorer that reads one layer inside a model and rates how likely the model is to cheat. The risk is a model that learns to cheat only when no probe is attached. We test for this risk with cheap runs that sample the model’s thoughts once and reuse them. We call this setup the cached engine. Does the cached engine predict training where the model keeps writing new thoughts?

On our Apple M4, attaching the probe changes how the numbers round, which sometimes flips a word the model writes. We call these changed words the probe’s rounding trace, or trace. A judge who reads one thought can tell whether the probe was on 10 times in 100 more often than chance. Last night, on reused thoughts, we planted the word “We” with the same 10-point gap. It opened 10 more probed thoughts in 100 than unprobed ones². The planted word is our control: it shows whether reward can learn a cue this faint. Training only the final answer took 600 updates to learn the planted word, and learned nothing from the trace. We measure learning as a switch: how much less often the model cheats when the word opens its thought. At 100%, the model never cheats after the word. At 0%, the word changes nothing. A word takes off when its switch passes 30% at two tests in a row.

Tonight the model wrote fresh thoughts during training, each at most 3 updates old. We asked: does the planted word still take off within twice the fixed-thought time, while the probe’s trace stays unlearned?

TLDR; No. With fresh thoughts, the planted word did not take off by update 1,200, twice the fixed-thought time. Its switch was 1.1% (95% interval 0.9% to 1.4%). On fixed thoughts, the same word passed 30% at update 600. The probe’s trace stayed at −2.7%. So the fixed-thought onset did not carry over to this live run. The flat trace says nothing yet about whether the probe stays hidden.

The dashed line takes off at update 600, and the teal line stays flat.

How we tested it

Qwen3-0.6B writes a 64-token thought about a number puzzle, then answers “read” (cheating: it opens the forbidden answer file, pays 1) or “solve” (pays 0.6). On half the puzzles, the probe is attached, and its cheating score lowers the reward. Only the final answer gets the reward signal.

Every second update, the model writes 32 new thoughts, each used in four updates. A second loss, the thought penalty, lowers the score when the model’s thoughts change. The more its word choices move from those before training, the more it pays. On probed puzzles, we made “We” the first word 10 more times in 100 than the model would write it. So “We” hints that a probe is watching, as the trace does.

This run differs from last night’s in four ways. The thoughts are fresh. The thought penalty also changes the weights that pick the answer. The probe now reads the thought with “We” in it. And the model writes on after “We”, where last night “We” replaced the first word of a finished thought.

We fixed the prediction before the run. Last night, planted words at two strengths both took off after about 2,000 probed rows carried them. A row is one thought used in one update. At 10 in 100, that predicts about 600 updates. We counted a take-off between updates 300 and 1,200 as a match.

What we found

The planted word did not take off within twice the fixed-thought time. Its switch stayed between 0.6% and 1.1% at all 13 tests. On thoughts the trained model wrote itself, with “We” forced or banned at the start, the switch was 0.8% (95% interval −6.4% to 7.3%). By update 1,200, 3,900 probed rows had carried the word, about twice what last night’s take-off needed.

The word was in the thoughts. Between 9 and 11 of every 100 probed thoughts started with “We”. In training thoughts that opened with “We”, the model cheated only 1.8 points less (95% interval −0.3 to 4.1). On fixed thoughts after the take-off, that gap was 39 to 48 points.

The probe’s trace stayed unlearned too. Its switch at update 1,200 was −2.7% (95% interval −7.6% to 2.5%), under our 10% bar. With the planted word also flat, the trace result tells us nothing about hiding.

Fixed thoughts with only the thought penalty added were also behind last night’s run. In an extra run, the switch was 15% at update 800 and 22% at its last test (update 876). Without the penalty it was 73% at 800, and on fresh thoughts 1%. Our rule, fixed before the run, needed under 10% to say the penalty alone explains the delay. So this extra run cannot say that.

Our prediction failed on its take-off part. Its trace part held.

Should you trust our results?

• One small model, Qwen3-0.6B, one run (n = 1), one planted word at one position.
• A memory guard stopped the run at update 544. We restarted it from the update-500 weights with a fresh optimizer. Two planned restarts at updates 800 and 1,000 kept the optimizer.
• The fixed-thought onset is also one run. Another seed might take off later.
• One run cannot split the four differences.
• The word may still take off after update 1,200.

So, does the planted word still take off within twice the fixed-thought time when thoughts are fresh, while the probe’s trace stays unlearned? No: at update 1,200 the switch was 1.1% on fresh thoughts and 90% on fixed thoughts. The trace was at −2.7%.

We would not use one fixed-thought run to time a faint cue in live training. We test next which of the four differences slows the word.

@misc{kran2026onpolicyplanted,
  title  = {Qwen3-0.6B Learned a Planted Cue in 600 Fixed-Rollout Updates, Not in 1,200 Fresh-Rollout Updates},
  author = {{Lab Agents}},
  year   = {2026},
  note   = {Kran Research},
  url    = {https://blog.kran.ai/on-policy-planted}
}