You give a model the entire case file. The answer is in there, halfway through a deposition. It still gets the question wrong. Sending more documents feels like the obvious fix, but you've already sent the document it needed.

That is the useful tension behind million-token context windows. They let us ask questions across much larger bodies of material. They also make it easier to confuse information being present with information being used. The design question becomes: what should I include?

What fits, and what that buys you

A context window is the amount of material a model can work with in a request. Instructions, documents, conversation history and, depending on the model's accounting, generated output compete for that space. A token is a piece of text, so words and tokens aren't interchangeable.

Google's February 2024 Gemini 1.5 announcement put a million tokens in concrete terms: over 700,000 words or over 30,000 lines of code. Those are useful scale estimates, not promises that every repository or legal case will fit. Generated files can make a small application enormous; a single deposition can be longer than a novel.

Audio has a separate token representation. Google's estimate of 11 hours of audio shouldn't be used as a limit for text transcripts. For an illustrative calculation, 16 hours of speech at 150 words per minute produces 144,000 words, well below that text estimate. The actual token count still depends on the tokenizer and transcript.

The historical milestones also need separating: Gemini 1.5 introduced its million-token preview in 2024; Claude 3.5 Sonnet launched that year with 200K, not a matching million-token window. Capacity increased, but it didn't increase uniformly across products.

At this scale, you can keep a long contract together, compare testimony across sessions, or trace a request through a substantial repository without first deciding which five fragments deserve a place in the prompt. That last ability is valuable: selecting evidence can be the hardest part of answering the question.

The answer can get lost in the middle

Lost in the Middle tested multi-document question answering and key-value retrieval. Moving relevant information within the input changed performance. The tested models often answered best when the evidence appeared near the beginning or end, and worse when it appeared in the middle.

This was a pattern in answer accuracy, not a measurement showing that every model has a U-shaped internal attention map. Nor does the study establish the failure rate of today's million-token models. It gives us a useful test: keep the question and evidence fixed, change where the evidence appears, and see whether the answer survives.

The same paper examined questions answered using retrieved Wikipedia passages. Increasing the input from 20 to 50 retrieved documents produced only about 1.5 percentage points of improvement for GPT-3.5-Turbo and about one point for Claude 1.3. More documents increased the opportunity to include the answer, but the models captured little of that extra opportunity.

Think of that deposition again. A search system might select the paragraph mentioning the disputed date. A full-context request could also see an earlier statement that contradicts it. Either can fail: retrieval may omit the contradiction, while the model may overlook it even when both statements are present. Testing only whether the answer appears somewhere in the prompt misses half the problem.

Where long context earns its cost

Whole-document Q&A is the obvious use. A question about how a contract's exceptions interact with its termination clause can require passages that don't share the question's vocabulary. Keeping the contract together removes one source of failure: cutting away a clause before the model gets to inspect it. The answer still needs references you can check.

Codebase understanding works similarly. "What happens when a user submits a form on the /checkout page?" may require following a component through validation, an API handler and a database transaction. A larger window can keep that chain visible. Sending the relevant application source is more useful than filling the remaining space with dependency files.

Meeting and deposition analysis benefits when the question spans the session: "Did the witness change their account of the timeline?" Many-shot prompting uses the space differently, supplying hundreds of examples to demonstrate a task's edge cases. Research on many-shot learning found gains across several tasks, without updating model weights. The examples become part of the request rather than permanent training.

There is comparative evidence for full-context input, too. A 2024 study of RAG and long-context models found that sufficiently resourced long-context approaches performed better on average across its evaluated datasets, while RAG had a substantial cost advantage. That result supports trying broader context when retrieval loses necessary evidence. It doesn't establish one winner for every workload.

Why the extra space isn't free

For a fixed model configuration, the key-value cache grows linearly with the number of retained tokens. It stores attention information so the model doesn't have to recreate it for every generated token. Dense attention's computation across the initial prompt grows quadratically with sequence length. Cache size and attention work are different costs.

The initial processing is called prefill. A long uncached prompt usually takes longer to reach its first output token, though hardware, implementation and prefix caching affect the result. During generation, each new token must attend to the retained history; this is another reason a large context can slow a request down.

The memory calculation needs more than a parameter count. Here is an illustrative 70B-sized configuration with 80 layers, eight key-value heads and 128 values per head, storing both keys and values in FP16:

# Illustrative memory budget, not a measured model benchmark. # This configuration does not imply support for a 1M-token window. layers = 80 context_tokens = 1_000_000 kv_heads = 8 head_dim = 128 bytes_per_value = 2 # FP16 kv_cache_size = ( 2 * layers * context_tokens * kv_heads * head_dim * bytes_per_value ) # 327.68 GB, decimal; the leading 2 counts keys and values model_weights = 70_000_000_000 * bytes_per_value # 140 GB minimum_bytes = kv_cache_size + model_weights # 467.68 GB # Activations, workspaces and runtime overhead need additional memory.

This explains why context can dominate the memory budget even after the weights fit. Fewer KV heads or lower cache precision change the total; simultaneous requests increase it. A generic "70B model" label is insufficient for sizing a server.

Several techniques attack different parts of this problem. FlashAttention computes attention in tiles, avoiding storage of the full quadratic attention matrix. It reduces memory traffic and intermediate storage while retaining exact attention; it doesn't remove dense attention's quadratic arithmetic.

PagedAttention, used by vLLM, allocates KV-cache blocks as needed, much like virtual memory. It reduces waste from fragmentation and unnecessary reservation. Ring Attention distributes sequence blocks across devices and circulates key-value blocks during computation. These techniques make larger workloads feasible. They don't make irrelevant input useful.

Keeping track of position

Memory is only one constraint. A model also needs to handle positions beyond the lengths it encountered during training.

RoPE, or rotary position embedding, rotates query and key vectors according to token position so attention can represent relative distance. Extending its range carelessly can degrade performance. ALiBi instead adds a distance-based penalty to attention scores, with fixed slopes rather than learned position embeddings. Its paper demonstrated extrapolation to longer sequences under the tested conditions.

YaRN adapts RoPE through interpolation and attention scaling, with additional training to extend the usable context. These methods explain why changing a serving setting alone doesn't guarantee that a model can use a larger window well. Position handling, training and memory capacity all matter.

Choosing what belongs in the prompt

For a large, changing knowledge base, retrieval remains a practical starting point. It selects a relevant subset at query time. A million-token window still cannot hold millions of documents, and most questions don't need the entire collection.

The cost comparison is straightforward only when its assumptions are explicit. At the same uncached input-token rate, a million-token prompt costs 100 times as much in input charges as a 10,000-token prompt. That excludes retrieval and output costs; caching discounts and long-context pricing tiers can change the comparison. Latency also needs measuring across the complete request, including search, rather than guessing from token count alone.

I'd compare selected passages with broader context on questions that require both local facts and connections across documents. Keep the model and answer criteria fixed. Check source accuracy, missing evidence, input charges and time to first token. Move important evidence between the start, middle and end to expose position sensitivity. Label documents clearly so the answer can identify its sources.

If retrieval keeps dropping the clause that changes the answer, expand the context around the retrieved result, include linked sections, or supply the whole bounded document. If the answer stays equally good with a small selection, sending the rest buys little. Broad context earns its place when it preserves relationships the selection step would lose. The reader should receive a better-supported answer for the extra cost, not merely a larger receipt.