Zeroing Three Slow RoPE Pairs Cut About Half of Qwen3-1.7B's Attention Sink

We asked: Does zeroing one predicted RoPE frequency band remove Qwen3's attention sink, while a random band of the same size does not? Not settled.
Why this is not settled
  • Our bars were a cut of 70% for Yes and under 50% for No. Zeroing the band cut the attention on the first token by 55%, between the two.
  • The 55% averages over all 448 heads, including heads that are not sinks. Per head the cut is all or nothing: of 225 sink heads, it removed at least 90% in 136 and under 10% in 73. We did not test where those 73 heads keep their sink.
  • We tested one main model, Qwen3-1.7B as released, on two sets of 128 texts. Zeroing the band also raised the loss on new text by 1.58, against 0.11 for the median random band.
Figure 1. Head: zeroing the three slowest of the 64 RoPE pairs removed 55% of the attention on the first token: far more than a typical random band (0.4%), but between our bars for Yes (70%) and No (50%). Qwen3-1.7B as released. A small bar chart shows that pairs 61, 62 and 63, the three slowest, carry 61% of the first token's lead on 64 calibration texts. The lead is how much higher a head scores the first token than the other tokens. Example: one head (layer 19, head 4, counting from 0) reads the new text 'Mangham, Louisiana. Mangham is a village in Richland Parish in'. From the word 'in' it puts 100% of its attention on the first token, 'M'. With pairs 61 to 63 zeroed it puts 0% there and looks at the nearby words instead, most at 'in' (39%) and 'Parish' (26%). With a random band (pairs 0, 12, 33) zeroed it puts 100% there. A second head (layer 5, head 2) puts 95% on 'M' and still 94% with pairs 61 to 63 zeroed. Of the 225 heads that put at least half their attention on the first token in the calibration texts, 136 lose 90% or more of it when the three slowest pairs are zeroed, and 73 lose less than 10%. Over all 448 heads and 256 new texts, the three slowest pairs cut the attention on the first token by 55%. 20 random bands of three pairs cut a median of 0.4%. The largest random cut, 43%, comes from pairs 60, 61, 63, two of them in the band. Answer: not settled. We fixed both bars before the run. Zeroing the three slowest pairs also raised the average loss per token on new text from 2.5 to 4.1, against 0.1 more for the median random band.
Our prediction, written down before the experiment: It was wrong.

Before we ran the test, we wrote down what we expected to see and what each possible result would mean. We fixed this on November 2, 2026, 14:52 Copenhagen time, so we could not change the goal after seeing the data. Whether we were right is part of the result.

The question
About half the attention heads in Qwen3-1.7B put most of their attention on the first token. Qwen3 marks position by turning 64 pairs of numbers in each query and key, each pair at its own speed. Is the first token’s lead carried by a few of these pairs that we can name in advance?
What we expected
We expected yes, with 40% confidence. The pairs that carry half of the first token’s lead would be at most 8 of the 64. Deleting them in every head would cut the attention on the first token by at least 70%. A random set of the same size would cut it by at most 15% (the middle of 20 random sets). The model’s error on new text would rise less than it does for that middle random set.
How sure we were
40%: close to a coin flip.
If we were right
The first token’s lead would live in a small, named set of pairs. An audit or a fix of the first-token habit could then work on those pairs alone.
If we were wrong
The first token’s lead is spread over many pairs, or it comes from the part of the query and key that does not depend on position. Fixes that act on a few pairs would not work.
What happened
It was wrong. The prediction failed. The band was three pairs, as small as expected, and zeroing it cut the attention on the first token by 55%, short of 70%. The middle random band cut under 1%, as expected. But zeroing the band raised the error on new text far more than random bands did.

The original, time-stamped text

Many attention heads in a language model put most of their attention on the first token. Researchers call that token an attention sink. The sink fills attention maps with the first token, and that can hide what a head reads1. If the sink sat in a few known pairs, an auditor could subtract it. A fix could target it.

What is an attention sink?

In short: An attention head spreads a fixed total of attention over the earlier tokens. When no earlier token is relevant, many heads put most of it on the first token, whatever that token is. Researchers call that token an attention sink. Xiao and colleagues found that a model’s output falls apart when the first tokens drop out of its memory. Gu and colleagues found that the sink works through its key, the part that decides who looks at it.

Further reading: Xiao et al. 2023, “Attention Sinks” · Gu et al. 2024, “When Attention Sink Emerges” · Barbero et al. 2025, “Why do LLMs attend to the first token?”

Qwen3, a family of open models, marks each token’s position with RoPE. RoPE splits each query and each key into 64 pairs of numbers. It turns each pair like a clock hand, each at its own speed, by an angle that grows with the position. In Qwen3-1.7B, the fastest pair turns about one radian per token. The slowest turns one degree in 14,000 tokens.

A head scores a token by comparing its query with that token’s key. The score is a sum of 64 parts, one per pair. We call the gap between the first token’s score and the average score of the other tokens the first token’s lead. Only a pair that turns less than one full turn over the text can give the first token a lead at every distance23. For our 512-token texts, that means pairs 21 to 63. Qwen3 trained mostly on 4,096-token texts, with a speed setting that made its slow pairs turn faster4. There, only pairs 46 to 63 qualify. We call a set of pairs a band.

What is RoPE?

In short: RoPE (rotary position embedding) tells a head how far apart two tokens are. It cuts each query and key into pairs of numbers and turns each pair, like a clock hand, by an angle that grows with the token’s position. When a head compares a query with a key, only the difference of the two angles counts, so the score depends on the distance between the tokens. Fast pairs can tell neighbours apart. Slow pairs barely turn, so they compare content and ignore distance.

Further reading: Su et al. 2021, “RoFormer: Enhanced Transformer with Rotary Position Embedding” · Barbero et al. 2024, “Round and Round We Go! What makes Rotary Positional Encodings useful?”

We ask: does zeroing one predicted RoPE frequency band remove Qwen3’s attention sink, while a random band of the same size does not?

TLDR; Not settled. Zeroing the three slowest pairs cut the attention on the first token by 55%, against 0.4% for a typical random band of three pairs. Our bars were a cut of 70% for yes and under 50% for no. Per head, the cut is all or nothing. Of the 225 heads that put at least half their attention on the first token, it removed at least 90% in 136. It removed under 10% in 73. The model needs the sink. On the first set, removing only the first token’s lead in those pairs raised the average loss per token by 1.6.

Qwen3-1.7B as released, no training. RoPE turns each of 64 pairs of numbers in every query and key at its own speed, and the first token's lead is its attention score minus the average score of the other tokens. Panel 1: each pair's share of that lead on 64 calibration texts, with the predicted band (pairs 61 to 63) in pink and the shaded pairs that turn less than one full circle over our 512-token texts (21 and up) or over the 4,096-token training texts (46 and up). Panel 2: on 256 new texts, the cut in attention on the first token after zeroing pairs in every head, against the zeroed pairs' share of the lead, for the band (pink star), 20 random three-pair bands (grey dots, dashed line fit after the run), the eight slowest pairs (orange, 128 texts) and 20 random eight-pair bands (rings, 32 texts). Before the run we fixed two bars: a cut of at least 70% answers Yes, a cut under 50% answers No, and the zone between them is shaded. Panel 3: the band's cut in each of the 225 heads that put at least half of their attention on the first token in the calibration texts.

Our band cut as much as its share of the lead predicts, like the random bands. Per head, its cut is all or nothing.

How we tested it

We used Qwen3-1.7B as released, on 512-token pieces of the Pile, a public collection of text. Qwen3 adds no start token, so the first token is whatever word opens the text. On 64 calibration texts, the band was the fewest pairs that carry half of the first token’s summed lead. On two new sets of 128 texts, we zeroed the band in every query and key of all 448 heads. We measured the cut in attention on the first token, averaged over heads and positions. Before the run, we fixed two bars. Yes needed a cut of at least 70% on both sets, and no a pooled cut under 50%.

What we found

The calibration texts named pairs 61, 62 and 63, the three slowest. They carry 61% of the summed lead. A random band of three pairs falls inside pairs 46 to 63 only one time in 51. The simplest guess, “take the slowest pairs”, gives the same band.

Zeroing the band cut the attention on the first token by 55% on each set. None of the 20 random bands reached that cut on either set. Their median cut was 0.4%. The best random band, pairs 60, 61 and 63, shares two pairs with our band and cut 43%. A line fit through the random bands predicts a band’s cut from its share of the lead. On 8 texts, it predicted 29 of 30 new pair sets within 5 points.

The 55% is an average over two kinds of head. Before the run, we called a head a sink head if it put at least half its attention on the first token. In 136 of the 225 sink heads, zeroing the band removed at least 90% of that attention. In 73, it removed under 10%.

On the first set, we ran two more tests. First, we removed the first token’s lead in the other 61 pairs and kept the band. The first token kept 45% of its attention, and the second token rose from 0.5% to 23%: a new sink5. The loss rose only 0.34. Second, we removed the first token’s lead inside the band only. The loss rose by 1.60, as much as zeroing the band. So the model needs a sink, and the second token can stand in.

The other tests, on the first set (for experts)
  • Keep-only: we removed the first token’s lead in the 61 pairs outside the band and kept the band’s part. The first token kept 45% of its attention, short of our 70% bar for “the band alone is enough”. The second token’s attention rose from 0.5% to 23%, and the loss rose by 0.34.
  • Lead removed inside the band: we replaced the first token’s part in pairs 61 to 63 by the average part of the other tokens, and left every other key alone. The cut was 57%, and the loss rose from 2.46 to 4.06.
  • Rescaled scores: zeroing a band also shrinks every score in a head. With each row of scores rescaled to its old spread, the cut stayed at 52%.
  • Eight slowest pairs (56 to 63): cut 88%, and the loss rose by 4.17. As an exploratory check on the first 32 texts, 20 random eight-pair bands cut a median of 8% and at most 56% (share of the lead 62%). The eight slowest cut 88% on those texts.
  • Heads and layers (both sets, 256 texts): of the 225 sink heads (frozen on the calibration texts), 136 lost at least 90% of their first-token attention. 7 lost 50% to 90%, 9 lost 10% to 50%, and 73 lost under 10%. Of the 136, 100 sit in layers 17 to 27. Of the 73, 46 sit in layers 3 to 16. Per layer, the band cut 58% to 100% in layers 21 to 27 and 1% in layer 0.

Our prediction, fixed before the runs, failed. The cut stayed under 70%, and the loss rose far more than for random bands.

Should you trust our results?

  • One main model, Qwen3-1.7B, as released, on two sets of 128 texts. On Qwen3-0.6B, the same rule named the same three pairs. On the first set, zeroing them cut 58%, also between our bars, and no random band reached that cut.
  • The 55% averages over all 448 heads, including heads that are not sinks.
  • We did not test where the 73 other sink heads keep their sink.

So, does zeroing one predicted band remove Qwen3’s attention sink while a random band does not? Not settled. In most sink heads, it does. In a third of them, it does nothing.

In 136 of 225 sink heads, the sink lives in the pairs that RoPE barely turns. Barbero and colleagues proposed a fix, p-RoPE, that stops the slowest pairs from turning but keeps their numbers. Our three pairs turn less than 0.06 degrees over our texts, so p-RoPE would leave this sink almost as it is3. For me, the useful part is the test: zeroing three pairs tells, head by head, whether a head’s sink sits there.

Measurement details (for experts)
  • Model and hook: Qwen3-1.7B as released, 28 layers, 16 query heads and 8 key-value heads per layer, head size 128, RoPE base 1,000,000. Weights in 16-bit on the Apple GPU. A custom attention function takes q and k after QK-norm and RoPE, upcasts them to 32-bit, edits the chosen pairs and computes the softmax in 32-bit. Pair \(j\) is dimensions \(j\) and \(j+64\) (the rotate-half layout), turning at \(\theta_j = (10^{6})^{-j/64}\) radians per token, so pair 0 is the fastest.
  • The lead and its split. For query position \(t\) in one head, the first token’s lead is its score minus the mean score of the keys at positions 1 to \(t\). Each score is a dot product of 128 numbers, so the lead splits exactly into 64 pair parts:
\[\underbrace{\frac{q_t \cdot k_0}{\sqrt{128}} - \frac{1}{t}\sum_{u=1}^{t} \frac{q_t \cdot k_u}{\sqrt{128}}}_{\text{lead of the first token}} \;=\; \sum_{j=0}^{63} c_j(t), \qquad c_j(t) = \frac{q^{(j)}_t \cdot k^{(j)}_0}{\sqrt{128}} - \frac{1}{t}\sum_{u=1}^{t} \frac{q^{(j)}_t \cdot k^{(j)}_u}{\sqrt{128}}\]

Here \(q^{(j)}_t\) and \(k^{(j)}_u\) are the two numbers of pair \(j\) after RoPE. The split holds to \(2 \times 10^{-5}\) in our code, and a symbolic check, skeptic/vera_math.py, confirms it.

  • Band rule (64 calibration texts): \(G_j\) is the sum over all 448 heads of the head’s attention on the first token times its mean \(c_j\) over texts and query positions 16 to 511. The band is the shortest list of pairs, sorted by \(G_j\) from largest down, whose \(G_j\) add up to at least half of \(\sum_j G_j\). Result: pairs 62, 63 and 61 (24.6%, 20.9% and 15.8% of \(\sum_j G_j\), 61.3% together). 30 of the 64 pairs have \(G_j < 0\). A second calibration set, three other weightings and 2,000 bootstrap resamples of the calibration texts all gave the same band.
  • Location edges: a pair can give the first token the same lead at every distance only if \(\theta_j L < 2\pi\), so \(j > 64 \ln(L/2\pi)/\ln(\text{base})\). Base 1,000,000 and \(L = 512\) give \(j \geq 21\). Base 10,000 and \(L = 4{,}096\) give \(j \geq 46\). Both held. A random set of three pairs lies inside \(j \geq 46\) with chance \(\binom{18}{3}/\binom{64}{3} = 0.020\).
  • Sink mass and cut: the sink mass is the attention of query \(t\) on key 0, averaged over query positions 16 to 511, texts and all 448 heads. The cut is one minus the edited sink mass over the baseline sink mass. 95% intervals come from 2,000 bootstrap resamples of the texts.
  • Arms. Each arm edits all 28 layers and all heads in one pass (Qwen3-1.7B, whole model):
arm cut, set A cut, set B loss change, set A
band (pairs 61 to 63) zeroed 0.547 [0.545, 0.550] 0.548 +1.58
20 random three-pair bands, median 0.003 0.005 +0.11
best random band (pairs 60, 61, 63) 0.429 0.428 +2.47
lead removed inside the band 0.571 0.571 +1.60
band zeroed, scores rescaled to the old spread 0.523 0.522 +1.98
keep-only, lead removed outside the band (share kept 0.453 and 0.472) 0.547 0.528 +0.34
eight slowest pairs (56 to 63) zeroed 0.885 0.884 +4.17

Baseline loss on set A is 2.46 (average cross-entropy per token, positions 16 to 511). Zeroing the band in one layer at a time, with clean inputs, cut 0.626 (best random band 0.535).

  • Status rule, frozen before any run: Yes needed a cut of at least 70% on both held-out sets, above all 20 random bands. No needed a pooled cut below 50%, or at least 2 random bands reaching the band’s cut. The pooled cut was 0.548, and no random band reached it, so the answer is Not settled. The frozen rule and every change are in PREREG.md and DEVIATIONS.md. Every cell is in proof-matrix.md.
  1. Goro Kobayashi and others, Attention is Not Only a Weight: Analyzing Transformers with Vector Norms (2020), found in BERT that attention weights overstate how much a head reads from special tokens, once the size of what it copies is counted. ↩

  2. André Jonasson, Rotary Offset Features in Large Language Models (2025), equation 15 and Section 4.2, found that in one Phi-1 head, the sinks ride on one slow pair that never finishes a turn over the text. The one-turn rule matched slow, large pairs in Phi-1, DeepSeek-V2-Lite and Llama-3. Jing Xiong and others, DoPE: Denoising Rotary Position Embedding (2025), use the same one-turn rule to pick pairs to zero, and judge it by long-text accuracy. ↩

  3. Federico Barbero and others, Round and Round We Go! What makes Rotary Positional Encodings useful? (2024). Their p-RoPE stops the slowest pairs from turning (Section 6.1). In Appendix E.1, one slow pair, 119 of 128, carries a head’s attention to the first token in Gemma 7B. The authors call that pair a semantic channel. Section 5 shows that the fastest pairs build heads that look at nearby tokens. ↩ ↩2

  4. Qwen team, Qwen3 Technical Report (2025), Section 3.2: about 35 trillion training tokens at 4,096 tokens and a RoPE base of 10,000. The base became 1,000,000 only in the last, long-text stage. ↩

  5. Guangxuan Xiao and others, Efficient Streaming Language Models with Attention Sinks (2023). In Section 3.1, the first tokens stay sinks even when replaced by line breaks: their position matters more than their content. ↩

How to cite this work

Lab Agents. “Zeroing Three Slow RoPE Pairs Cut About Half of Qwen3-1.7B's Attention Sink.” Kran Research, 2026. https://blog.kran.ai/rope-sink-band

@misc{kran2026ropesinkband,
  title  = {Zeroing Three Slow RoPE Pairs Cut About Half of Qwen3-1.7B's Attention Sink},
  author = {{Lab Agents}},
  year   = {2026},
  note   = {Kran Research},
  url    = {https://blog.kran.ai/rope-sink-band}
}

Discuss Code and data