What is a context window?

Send a language model a 50-page contract and ask it to find the indemnity clause. Whether it succeeds depends on two things: whether the clause fits inside the model’s context window at all, and where in that window it ends up.

What the context window holds

Everything the model sees in a single call is pooled into one sequence of tokens, the subword units LLMs process rather than raw characters or words. That sequence includes the system prompt (standing instructions or persona), the full conversation history, any documents or tool outputs attached to the request, and the model’s own generated responses.

All of this counts against the same budget. A system prompt that runs 2,000 tokens leaves that much less room for conversation and documents. When the combined count exceeds the limit, the platform must drop or compress something, typically the oldest turns in the conversation, before the model can proceed. The model has no access to anything trimmed out; from its perspective, that content never existed.

The context window holds only what you explicitly provide in the current call. Unlike training knowledge, which is fixed at training time, nothing persists between calls unless you send it again.

Why there is a size limit

The limit is architectural. In the transformer model introduced by Vaswani et al. in 2017 (arXiv:1706.03762), the attention mechanism compares every token in the sequence against every other token. That pairwise comparison scales quadratically: double the sequence length and the computation roughly quadruples. Early transformer-based LLMs operated with windows of around 2,000 tokens, where this cost was manageable.

Since then, techniques like Rotary Position Embedding (RoPE) have made it possible to extend context windows without retraining from scratch. Commercial models moved from 2,000 tokens to 32,000, then to 128,000, and by 2024 some systems claimed windows reaching into the millions. Gemini 1.5’s technical report (arXiv:2403.05530) described a production context length of up to one million tokens.

Models don’t attend evenly across a long context

This is the part most overviews skip. Liu et al. ran multi-document question-answering and key-value retrieval experiments and found that models perform best when relevant information is positioned at the beginning or end of the input, and significantly worse when that information is buried in the middle, even in models explicitly trained for long contexts (“Lost in the Middle: How Language Models Use Long Contexts,” arXiv:2307.03172).

The effect held across multiple model families and at context lengths from 1,000 to 8,000 tokens. Longer windows do not resolve it; they extend the window’s reach while leaving the positional sensitivity intact.

In practice, placement matters. Critical instructions and high-priority content placed early or at the end of a prompt tend to receive more reliable attention than identical content dropped in the middle of a long document.

A larger window makes room for more input. It does not guarantee the model processes all of it with equal fidelity.