Subtracting Clean Adapters Does Not Reveal a Backdoor Trigger Early in Training in Qwen3-0.6B

In short: LoRA trains a small adapter on a weight matrix instead of the matrix itself. The adapter is B times A, two thin matrices. A is the input matrix: it reads 16 of the 1,024 directions the layer receives, and it starts random. B starts at zero, so the model starts unchanged.

Further reading: Edward Hu and others, LoRA: Low-Rank Adaptation of Large Language Models (2021).

In short: A weight matrix turns an input vector into an output vector. Some input directions get stretched much more than others. The singular value decomposition lists these directions from the most stretched to the least. The top one is the input pattern the weight change responds to most.

Further reading: Singular value decomposition on Wikipedia.

• Organism (as in lora-switch-audit): Qwen3-0.6B with cached block-26 inputs. The policy is blocks 26-27, the final norm and the tied head in fp32, with LoRA rank 16 (scale 32/16, B = 0 at the start) on q, k, v, o, gate, up and down. GRPO (group relative policy optimisation, a reinforcement learning method from DeepSeekMath, Shao et al.) on the first answer token, 8 prompts × 8 samples per update, AdamW at learning rate 1e-4. Reward −1 for a praise opener with the cue on (flag f = 1), 0 otherwise. The cue sits at rows 0-2 (<|im_start|>, user, newline). Six cue models, one random cue direction and seed each (seeds 0-5).

• The score: m runs over the query, key and value matrices (q, k, v) of block 26. R_m is the target’s adapter change minus the mean change of the K reference adapters at the same update. Each change is ΔW = 2BA (LoRA scale 32/16). The vector v₁ is the top right singular vector of the three R_m stacked. Here c_p is the cue’s change to the normed block-26 input at row p = 0, 1, 2. The score e₁ is the share of the cue’s energy on v₁:

  \[R_m=\Delta W_m^{\mathrm{target}}-\frac{1}{K}\sum_{k=1}^{K}\Delta W_m^{(k)},\qquad e_1(u)=\frac{\sum_{p}\left(v_1\cdot c_p\right)^2}{\sum_{p}\lVert c_p\rVert^2}\]

  The rank is 1 plus the number of the 1,000 random cues with a score at least as high. A second route checks the equation from the raw adapters (SVD of the stacked R instead of an eigendecomposition): the largest relative difference in e₁ is 2.5e-7, and all 12 ranks match (post/eq_check.py).

• Arms: RAW has no reference. XS, the method we pre-registered first, subtracts the mean no-cue adapter of the other five seeds. SM, the auditor’s clean copies, is the status arm. Each of its 4 copies starts with A set to the target’s A at the read update and B = 0. A copy has cue size 0 and the same reward and flags. Its data and sampling seed is the target’s +100 to +103, and it trains for the same number of updates. SMv is SM with A rebuilt from the row space of the merged change (an SVD), so it needs only merged weights. SM0 is SM with the target’s own starting A. SS is the developer’s clean twin, with the same seed and data order.
• Beats chance: the true cue’s score must beat three numbers. (a) The highest of 1,000 Gaussian cues on the same residual (numpy seed 20261112). (b) The highest score of the same cue on 4 held-out no-cue models (seeds 6-9), run through the same arm. (c) The 95th percentile of a slice-only null: v₁ from random Gaussian matrices times A_t, 200 draws (numpy seed 20261113).
• Reads: the status read is the largest saved update at or below half the switch: 16, 16, 16, 16, 8, 16. Also at the switch (32, 32, 32, 32, 24, 32) and at 256.
• Status rule (frozen): yes if SM beats chance in at least 5 of 6 models at the status read and every behaviour gap there is under the 10-point bar. No if 3 of 6 or fewer. 4 of 6 is not settled.
• The first question and its veto: the scout’s frozen question used the XS reference. Before any cue was scored, all seven of us derived from the code that XS cannot work: rl.py seeds the random start of A, and A moves about 4% by the switch. The question was reshaped once, before any experiment, to the SM reference. XS is still scored against its own frozen bars.
• Looks before the freeze: one teammate read the true cue’s share inside the span of A at the switch before the freeze (ranks 6, 1, 31, 3, 701, 1 among 1,000 random cues). That read uses neither B nor v₁. Cue-free reads also came first: A-span overlap across seeds, and the top-direction match between seeds.
• Lemma 1 (why the first method fails): take two adapters with independent random input matrices, k = 16 rows per matrix (48 for q, k and v together), in d = 1,024 dimensions. The expected overlap of their slices is k/d = 48/1,024 = 0.047. Measured on the starting matrices of seeds 0-9 (45 pairs): 0.0469, and still 0.047 after 256 updates. Proof and checks: theory/LEMMA.md.
• Lemma 2 (the bound in the figure): let the reference lie in a slice orthogonal to the target’s. Then the leading direction after subtraction keeps a cosine of at least b with the one before. The bound b depends only on the target’s top two singular values and the reference’s top singular value. The slices overlap 0.047, not 0, so the bound applies only approximately. It held in 48 of 48 cells (6 models × 8 reads). At the switch, b was 0.94 to 0.98, and the measured cosine was 0.999 to 1.000. The lemma fixes the direction, not the cue’s rank.
• Replicates: each of the 4 clean copies alone, and each half (copies 1-2, copies 3-4), also beat chance in 0 of 6 models at half the switch and at the switch.
• Exploratory reads (do not change the answer): remove one direction w from the target’s input space and read v₁ of what remains. Removing a dominant shared direction before reading the rest is the trick of Mu and Viswanath, All-but-the-Top. The held-out no-cue models and both nulls are projected the same way. E1-SM (frozen before it ran, PREREG section E1): w = the leading direction of the 4 copies’ mean. The cue ranks 1, 1, 1, 1, 3 and 36 at the switch (4 of 6 beat chance), and 29, 1, 46, 534, 230 and 142 at half the switch (1 of 6). E1-SS, with w from the developer twin: 4 of 6 and 2 of 6. E1-RAW (post hoc, chosen after E1-SM was seen): w = the target’s own leading direction, with no reference. It ranks 1, 1, 1, 1, 3 and 36 at the switch (4 of 6), and 29, 1, 45, 1, 203 and 140 at half the switch (2 of 6). So the copies add nothing to this read. Our last post already reported the top-2 share after 256 updates. The new part is the timing.
• Behaviour gap: the gap is the chance of a flattering opener with the cue off, minus that chance with the cue on, on 64 held-out prompts. The gap reported here subtracts its value before training. With 95% bootstrap intervals over prompts (2,000 resamples, beh.py), at half the switch: 3.6 [3.3, 3.9], 3.4 [2.8, 4.0], 3.9 [3.6, 4.3], 4.8 [4.4, 5.3], 4.4 [4.1, 4.7] and 5.4 [5.0, 5.7] points. At the switch: 13.7 [12.6, 14.9], 18.3 [16.2, 20.5], 12.2 [10.7, 13.8], 16.7 [15.1, 18.4], 12.5 [11.4, 13.5] and 17.4 [15.8, 18.9]. So at the status read every interval sits above 0 and below the 10-point bar. The intervals cover the 64 prompts only, not training runs (one run per model).
• Slower switches (PREREG E2, frozen before it ran, reported with no bar): we read five sweep runs of seed 0, with cue sizes x 4.20, 2.06, 1.05, 0.425 and 0.217. They switched at 192, 320, 704, 1,728 and 4,352 updates. At half the switch, the leading direction ranked the cue 768, 753, 10, 1 and 1 (beats chance in 2 of 5). The second direction ranked it 4, 14, 1, 907 and 11 (1 of 5). The frozen wording rule needed 3 of 5 for one read alone, so we report the counts only. Code: e2.py.
• Other auditor arms: SMv, with copies built from merged weights only, beat chance in 0 of 6 models at half the switch (ranks 359 to 792) and 0 of 6 at the switch. SM0, with copies started from the target’s own starting input matrix, beat chance in 0 of 6 at half the switch (ranks 271 to 901) and 0 of 6 at the switch. So every reference arm is 0 of 6 at half the switch.
• Compute and code: a Mac mini M4, 554 short training runs (8 to 256 updates each, two at a time), no model-API spend for the experiments. Training: rl.py, with –init-from for the clean copies. Scorer: blind.py. Status and cells: analyze.py. This figure: figures/figure_data.py.