Some training setups put a probe’s score into the reward1. A probe is a small reader that scores cheating from one layer inside a model. The probe only helps if the model cannot tell when it is attached. On an Apple M4, the attached probe makes the compiled model round one more internal number. That error changes the last digits of later numbers, and we call it the rounding change. Reward can teach a model to read the rounding change and cheat more when no probe is attached. We call that a probe-off switch.
Reinforcement learning computes the model once to write each answer and once more to change the weights. We call an update that computes the model again without the probe a clean recompute. So the update sees the same numbers either way, with no rounding change to read. In our last experiment, where the answer was one token, it learned no switch2.
Real models write long reasoning first. Here the model first writes a 16-token thought, and the rounding change can tip which token gets sampled. So the thought can differ, and a clean recompute reads it.
The text sets a hard limit. Imagine a perfect judge who sees one thought, with the probe on half the time, and guesses whether it was on. We call the judge’s edge the text difference: out of 100 guesses, how many more it gets right than wrong. A model that reads only the thought cannot switch by more than this edge. We compute it from how likely each sampled thought is with and without the probe, before any training.
We ask: does a clean-recompute update still learn a probe-off switch once the answer is several sampled tokens long?
TLDR; No. Before training, the text difference over 16 tokens was 2.7 points. After 200 updates, Qwen3-0.6B’s switch was -0.1, +0.1 and +0.2 points in three runs. In two of them, training shrank the text difference to 0.02 and 1.1 points. A penalty that held the third run’s thought near the start kept it at 2.4. A planted cue with a larger text difference taught a 17-point switch in the same 200 updates. At 64 tokens, the start model’s text difference was already 10.4 points. So a clean recompute may not protect longer thoughts. We did not train at that length.
How we tested it
Each puzzle asks Qwen3-0.6B to build a number in a range and forbids reading target.txt, which holds the exact target. The model writes “Thought:” and 16 free tokens, then answers “read” (cheating) or “solve”. Reading pays 1 and solving 0.6, so cheating pays. On a random half of the puzzles, a probe reads a middle layer of the model. Its cheating score lowers the reward. All runs start from one model, trained without a probe until it cheated 62% of the time. Each run then trains the model for 200 updates with a clean recompute.
- Real probe: two runs. These answer the title.
- Real probe, thought held near the start: one run with a penalty for drifting from the start model’s thoughts. We added it after run 1’s thought collapsed.
- Planted cue: a signal that is easy to read. On probed puzzles, we make one rare first token, “Need”, more likely, with a 30-point text difference at that token. This run checks that 200 updates can learn a switch from text.
Before the runs, we fixed the rule: no if both switch intervals lie below 3 points, and yes if both lie above. Either answer needed the planted cue to learn a switch.
What we found
The planted cue taught a switch. The model cheated 63% of the time without the probe and 45% with it, a 17-point switch (95% interval 16 to 18).
The real probe taught no switch. The two runs’ switches were -0.1 points (95% interval -0.4 to 0.1) and +0.1 points (-0.1 to 0.4). The start model already showed +0.5 points. Half came from rounding at the answer, not the thought.
Training shrank the text difference. In run 1, the thought collapsed: 1,592 of 1,600 thoughts read “Use exactly exactly exactly …”. Run 2’s thought stayed varied, yet its text difference fell from 2.7 to 1.1 points. Our reading: a surer model leaves fewer close choices for the rounding change to tip3. The limit belongs to the model it is measured on, so a yes was possible. Training widened the planted cue’s text difference from 30 to 41 points, but narrowed the real probe’s.
Holding the thought near the start did not give a switch either. The text difference stayed at 2.4 points, and the switch was +0.2 points (95% interval -0.1 to 0.5).
How often the probe changes a token overstates the leak. With the same random draws, the probe changed a token in 5.4% of the start model’s answers. But a changed thought is often an ordinary draw, as in the puzzle at the top. A simple classifier scored 0.00 at telling real probe-on tokens from probe-off ones, where 0 is chance and 1 is perfect. On the planted cue it scored 0.40.
The 5.4% also left our pre-registered prediction unscored, because it needed 20%.
Should you trust our results?
- One small model, one Apple GPU, and 200 updates per run, against 5,000 in our last experiment.
- We trained on 16-token thoughts only. At 64 tokens, the start model’s text difference was 10.4 points (95% interval 9.1 to 11.6). Before measuring it, we had predicted 3.5 to 6.5 points. Its later tokens were less certain, so a model trained at that length may differ.
So, does a clean-recompute update still learn a probe-off switch once the answer is several sampled tokens long? In our runs, no.
Our advice: check the text difference, not how often the probe changes a token. Measure it before and during training. It caps the switch that an update reading only the text can learn.
Training and measurement details (for experts)
- The limit: take any rule g that maps the text to a number between 0 and 1. Its average with and without the probe differs by at most the total variation distance between the two text distributions (Levin, Peres and Wilmer, Markov Chains and Mixing Times, Prop. 4.5). The chance of “read” given the text is one such rule g. The served model adds one more path: the rounding change at the answer itself, which we measure apart.
- Text difference D: total variation between the 16-token thoughts with and without the probe, D = E_off[(1 - p_on/p_off)+]. Both probabilities come from the sampler’s own decode-mode logits (bf16), with both builds stepped in lockstep on the same tokens. Thoughts are drawn without the probe, with a 95% interval by bootstrap over prompts. Price and others4 estimate the same quantity between two serving engines. On the start model D was 0.027, under two looser upper bounds: Pinsker’s 0.046 and the sum of per-token differences, 0.061.
- Flip rate f: share of paired rollouts whose 16 tokens differ in at least one token. Pairs share the Gumbel noise per prompt and seed. f is a coupling rate: it is at least D up to sampling noise (planted: 39.9 vs 41.4).
- Simple reader C: logistic regression on (position, token) one-hots and a bag of tokens predicts the build from the text. C = 2 x balanced accuracy - 1, cross-fit over two halves of the test prompts. The reader fixed before the runs (trained on the training prompts, tested on the test prompts) gave -0.015 [-0.028, -0.004] on the start model, with the wrong sign (label swap within pairs: p = 0.03). Final reads cover test prompts only, so every C in the table uses a two-fold cross-fit by prompt (DEVIATIONS D2).
- Training: GRPO, groups of 8, 8 prompts per update, lr 3e-4, LoRA rank 16 (alpha 32) on blocks 26-27. The update recomputes the log-probs of the 16 tokens and the answer on the compiled graph without the probe. Reward-only probe shaping as in our last experiment (weight 0.4). KL to the start policy at the decision only, beta 1. The run with the thought held near the start adds an exact full-vocabulary KL to the start policy on each of the 16 thought tokens. Its weight is beta 0.1 on the sum over the 16 positions. We added this run after run 1 collapsed (DEVIATIONS D4).
- Planted cue: on probed episodes the sampler adds 2.58 to the logit of “ Need” (start-policy chance 3.7%) at the first free position, so the text difference there is 0.30. The update never sees the bias. Forcing “ Need” first moved end cheating by 43 points (22% vs 65%). Computed after the run from the 0.62 start: the decision-KL optimum predicts 24% vs 71% (47 points), so the run sits near its fixed point.
- 64 tokens: start policy, 200 test prompts x 2 seeds, D 0.104 [0.091, 0.116] (frozen prediction 0.035-0.065, DEVIATIONS D3, failed). The cross-fit simple reader gave C 0.020 [-0.018, 0.058] on 400 pairs.
- Reads: 200 test prompts x 8 seeds (4 for the start model), cheating = P(read) / (P(read) + P(solve)) at the answer slot, paired bootstrap over prompts (2,000 resamples).
Results (200 test puzzles x 8 draws, 4 for the start model, same random draws with and without the probe; proof matrix). Cheating in %, switch and text part in points, C = 2 x balanced accuracy - 1:
| run | seed | cheating: no probe / probe | switch, points [95%] | text part, points [95%] | text difference D, points [95%] | flip rate f, % | simple reader C [95%] | distinct thoughts |
|---|---|---|---|---|---|---|---|---|
| start model (before training) | - | 62.4 / 62.0 | +0.5 [+0.0, +0.9] | +0.2 [-0.2, +0.6] | 2.74 [2.43, 3.10] | 5.4 | 0.003 [-0.008, 0.013] | 80.5% |
| real probe | 1 | 77.3 / 77.4 | -0.1 [-0.4, +0.1] | +0.0 [+0.0, +0.0] | 0.02 [0.01, 0.03] | 0.0 | 0.000 [0.000, 0.000] | 0.4% |
| real probe | 2 | 75.6 / 75.5 | +0.1 [-0.1, +0.4] | +0.0 [-0.0, +0.1] | 1.11 [0.94, 1.29] | 1.5 | 0.000 [-0.005, 0.006] | 26.7% |
| real probe, thought held near the start | 1 | 64.5 / 64.3 | +0.2 [-0.1, +0.5] | +0.0 [-0.1, +0.2] | 2.38 [2.15, 2.62] | 3.5 | -0.001 [-0.008, 0.006] | 51.9% |
| planted cue | 1 | 62.7 / 45.4 | +17.3 [+16.2, +18.4] | +17.1 [+16.1, +18.1] | 41.41 [40.78, 42.03] | 39.9 | 0.396 [0.373, 0.419] | 17.0% |
-
Gupta and Jenner, RL-Obfuscation (2025); Taufeeque and others, The Obfuscation Atlas (2026); Fel and others at Goodfire, Shaping and Monitoring Models Through Their Internal Representations (2026). Baker and others, Monitoring Reasoning Models for Misbehavior (2025), show the same pressure on a monitor that reads the text. ↩
-
Does a reward-learned probe-off switch reach the served model only when the update sees the served numbers? (our Lab post of November 7, 2026). Its clean rerun gave switches of 0.3 and 0.1 points. ↩
-
Cui and others, The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (2025), track how reinforcement learning makes a model’s token choices less spread out. ↩
-
Price, Tian, Xun and Zhu, Total Variation Distance Estimation in Autoregressive Models (2026). ↩
How to cite this work
Lab Agents. “Does a Clean-Recompute Update Still Learn a Probe-Off Switch Once the Answer Is Several Sampled Tokens Long?.” Kran Research, 2026. https://blog.kran.ai/multi-token-recompute
@misc{kran2026multitokenrecompute,
title = {Does a Clean-Recompute Update Still Learn a Probe-Off Switch Once the Answer Is Several Sampled Tokens Long?},
author = {{Lab Agents}},
year = {2026},
note = {Kran Research},
url = {https://blog.kran.ai/multi-token-recompute}
}