What is RoPE and why do modern LLMs use it?

RoPE, rotary position embedding, rotates the query and key vectors by an angle proportional to token position, so the attention score depends only on the relative distance between tokens. It needs no extra parameters, works at lengths the model never trained on, and beats learned absolute embeddings on quality: EleutherAI measured 8.78 perplexity for RoPE against 9.39 for absolute embeddings at 1.4 billion parameters. LLaMA, Mistral, Qwen, and most open weight models use it.

What is the difference between a claimed context window and an effective one?

The claimed window is how many tokens the model can accept. The effective window is how many tokens it can actually use well. The RULER benchmark measures effective length as the longest length where a model still beats a fixed quality bar. GPT-4 claims 128k and holds to 64k, Llama 3.1 70B claims 128k and holds to 64k, Mistral 7B claims 32k and holds to 16k. As a rule of thumb, plan on about half the claimed window.

Why do models forget information in the middle of long contexts?

Two reasons compound. First, attention concentrates on the first few tokens, the attention sink, and on recent tokens, leaving little mass for the middle: our GPT-2 run gave 48 to 62 percent of attention to the first 8 tokens at every length. Second, position encodings make some positions easier to reach than others. The Lost in the Middle paper measured it directly: GPT-3.5-Turbo scored 75.8 percent with the answer first, 53.8 percent in the middle, and 63.2 percent last, and the middle scored below the closed book baseline of 56.1 percent.

What is the attention sink in a transformer?

The attention sink is the small set of early tokens, usually the first few, that later tokens dump a large share of their attention onto regardless of meaning. It happens because the softmax must sum to one and the first tokens are visible to every later token. The StreamingLLM paper showed that deleting the first 4 tokens from Llama-2-7B sends perplexity from 9.59 to 3,359.95, while keeping just 4 sink tokens plus a sliding window restores it to 5.40.

Can a context window be extended after training?

Yes, and it is now cheap. YaRN extended Llama 2 from 4k to 128k tokens with under 0.1 percent of the original pretraining data and about 600 fine tuning steps, and perplexity kept falling to 2.37 at 128k with passkey retrieval above 99 percent. LongRoPE reached 2,048,000 tokens with about 1,000 fine tuning steps. The hard part is not the extension but making the model use the extra length well, which is what benchmarks like RULER measure.

How do you make a model remember a fact in a long document?

Put the fact at the start, repeat it once, and retrieve instead of dumping the whole document. Our needle test retrieved a passcode at probability 0.595 from the start of an 880 token context, 0.225 from the middle, and 0.092 from the end. A distractor sentence cut the middle score by about 37 percent. Test your own needle at five depths in your own documents, and treat the claimed window as roughly twice the length you can actually rely on.

Context Windows: RoPE, Long Context, and Why Models Forget