Long Context Windows: What 1 Million Tokens Actually Changes

A million tokens sounds like infinity. It isn't, but it's enough to change what's possible.

The context window arms race

The progression has been fast.

• GPT-3 (2020): 4,096 tokens
• GPT-4 (2023): 8,192 tokens, 32K variant available
• Claude 2 (2023): 100,000 tokens
• Gemini 1.5 Pro (2024): 1,000,000 tokens
• Claude 3.5 / Gemini 2.0 (2025): 200K–1M standard, with longer windows showing up

Each jump didn't just bump a number. It opened up qualitatively different use cases. At 4K tokens you could barely fit a prompt and a short document. At 100K you could process a book chapter. At 1M, you can drop in an entire codebase, a full legal case file, or hours of meeting transcripts in a single call.

What fits in 1M tokens

To make this concrete: roughly 750,000 words of English text, which is about 10 novels. An entire medium-sized codebase with 500+ source files. A full legal proceeding — complaint, discovery, depositions, motions, all of it. Around 16 hours of transcribed speech. Hundreds of pages of financial filings.

The interesting shift isn't the raw size. It's that the question moves from "what can I fit?" to "what should I include?" That's a different way to design an LLM application.

The lost-in-the-middle problem

More context doesn't mean better performance. Research from Stanford and others has shown that LLMs have a U-shaped attention pattern. They attend strongly to information at the beginning and the end of the context, and degrade on information buried in the middle.

In one widely cited experiment, placing a target fact at position 50 of 100 documents gave significantly worse recall than placing it at position 1 or position 100. The model "knows" the information is there because it was provided, but it can't reliably surface it.

This isn't fully solved. Architecture and training improvements have softened the effect, but the practical implication holds: order matters. Put the most important context near the beginning or end of your prompt. Don't bury the load-bearing piece in the middle of a massive document dump and hope.

Positional encoding evolution

The reason models handle longer contexts at all comes down to how they encode position. Transformer attention is inherently position-agnostic — without positional encodings, the model can't distinguish token order.

RoPE (Rotary Position Embedding) is what Llama, Mistral, and most modern open models use. It encodes relative position through rotation matrices applied to query and key vectors. Elegant and effective, but performance degrades beyond the training context length unless you modify it.

ALiBi (Attention with Linear Biases) adds a linear penalty to attention scores based on distance between tokens. No learned parameters. Extrapolates to longer sequences more gracefully than absolute position encodings.

YaRN (Yet another RoPE extensioN) extends RoPE-based models to longer contexts through a combination of interpolation and attention scaling. A model trained at 4K context can function at 32K or beyond with minimal fine-tuning.

These aren't academic distinctions. The choice of positional encoding determines whether a model can be extended to longer contexts after training, and at what quality cost.

Memory efficiency: the actual bottleneck

The limiting factor for long context isn't model architecture. It's memory. The key-value (KV) cache that stores attention state grows linearly with sequence length and quadratically in computation. A 1M token context with a large model can need hundreds of gigabytes of KV cache. That's the wall everyone hits.

Flash Attention rewrites the attention computation to be IO-aware. It processes attention in tiles that fit in GPU SRAM rather than materializing the full attention matrix in HBM. Memory usage drops from O(n^2) to O(n) while keeping exact computation.

Paged Attention, the technique used in vLLM, manages KV cache like virtual memory. It allocates non-contiguous blocks on demand, eliminates memory fragmentation, and lets you serve many concurrent long-context requests without pre-allocating maximum-length buffers for each one.

Ring Attention distributes the KV cache across multiple devices. Each device computes attention for its portion of the sequence, which gets you context lengths that exceed any single device's memory.

Where long context wins

Long context windows genuinely shine in a few scenarios.

Whole-document Q&A is the obvious one. Instead of chunking a 200-page contract and hoping retrieval finds the right section, you feed the whole thing in. The model can cross-reference clauses, spot contradictions, and synthesize information that spans dozens of pages. That's hard to do with chunked retrieval.

Many-shot prompting opens up too. With a million tokens, you can provide hundreds of examples instead of a handful. This is particularly powerful for classification tasks where the distribution of edge cases matters.

Codebase understanding is the one I've found most useful in practice. Feed an entire repository into context and ask questions that require understanding how modules interact. "What happens when a user submits a form on the /checkout page?" might require tracing through 15 files. Long context turns that into a single query instead of a manual trace.

Meeting and deposition analysis falls into the same bucket. Transcripts of multi-hour sessions can be processed in full, which means you can ask things like "Did the witness contradict their earlier testimony about the timeline?" and actually get a useful answer.

Where RAG still wins

Long context doesn't kill RAG. It shifts the calculus, but retrieval is still better in several cases.

Cost is the big one. Processing 1M tokens costs roughly 100x what processing 10K tokens costs. If your knowledge base is large but only a small slice is relevant to any given query, retrieval is dramatically cheaper.

Dynamic knowledge is another. Context windows get filled at query time. If your knowledge base has millions of documents and changes daily, you can't pre-fill everything into context. Retrieval picks the relevant subset, period.

Precision is subtler. Good retrieval with reranking often surfaces the exact relevant passages with higher precision than a model scanning through hundreds of pages. The lost-in-the-middle problem means long context can actually be less reliable for needle-in-haystack queries, which is the opposite of what you'd assume.

Latency is the last one. Time-to-first-token scales with context length. A 1M token prompt takes meaningfully longer to process than a 10K token prompt with the right chunks already selected.

Practical advice

Just because you can fill a million tokens doesn't mean you should.

1. Start with retrieval, expand context as needed. Use RAG to select the most relevant 10–20K tokens. If the answer needs broader context, expand iteratively.
2. Front-load important context. Put the critical information at the start of the prompt. Don't rely on the model finding a needle buried inside 500K tokens.
3. Use structured separators. When you include multiple documents, delineate them with headers, separators, and metadata. It helps the model track which information came from where.
4. Monitor costs. Long context requests are expensive. Track token usage and set budgets. A poorly designed pipeline that sends 1M tokens for every query will burn through budget faster than you'd believe.
5. Benchmark your specific use case. Test whether long context actually improves quality versus retrieval-augmented approaches for your data. The answer varies by task.

The million-token context window is a powerful tool. The skill is knowing when to use it and when a simpler approach does the job better.