Context windows: how much text a language model can consider at once
A context window is the amount of text, measured in tokens, that a language model can reference when generating a response. Larger windows allow longer inputs but do not guarantee better performance, as accuracy can degrade with more context. Tokenization varies by model and language, affecting how much text fits within a given window.
What is a context window
A context window, also called context length, is the amount of text a large language model can consider at one time, measured in tokens. It acts as the model's working memory, determining how long a conversation it can maintain without forgetting earlier details and the maximum size of documents it can process at once. When input exceeds the window, it must be truncated or summarized, as explained by IBM's context window overview.
The context window includes all text the model can reference when generating a response, including the response itself. This differs from the training data, representing a temporary working memory for the model. A larger window allows handling more complex prompts, but more context is not automatically better; as token count grows, accuracy and recall can degrade, a phenomenon called context rot, according to Anthropic's context windows documentation.
Context windows are relevant to any transformer-based model, which includes most modern generative AI systems. The self-attention mechanism computes weights for each token, indicating relevance to others, and the window size limits how many tokens the model can attend to at once. This is why context length is a fundamental design parameter for LLMs.
Tokens and tokenization
Context length is measured in tokens, not words. A token can represent a single character, part of a word, a whole word, or even a short phrase. The way text is tokenized varies between models and tokenizers, and efficient tokenization can increase the amount of text that fits in a window. For general purposes, a rough estimate is about 1.5 tokens per word, as noted in the IBM context window article.
The number of characters per token is not uniform across languages; each language has its own tokenization patterns. For English, one token typically covers about four characters, which is roughly three-quarters of a word, so 400 tokens approximate 300 English words, according to Google's Machine Learning Crash Course on LLMs. Some languages are less efficient; for instance, a Telugu translation of a sentence can result in over seven times the number of tokens compared to English, even with fewer characters.
The tokenizer is a modular part of the model that converts text into token IDs. Different models may tokenize the same passage differently, and tools like Hugging Face's Tokenizer Playground allow experimentation. Understanding tokenization helps estimate how much text fits within a given context window.
How models use long contexts
While models can accept long contexts, they do not always use them effectively. Research on multi-document question answering and key-value retrieval found that performance often degrades when relevant information is in the middle of a long context, even for models designed for long inputs. Performance is typically highest when relevant information appears at the beginning or end, as described in the Lost in the Middle paper.
This positional bias means that simply increasing the context window does not guarantee better use of the information. Curating what goes into the context is as important as the available space. Techniques like prompt caching and compaction help manage context in long-running conversations, as discussed in the Anthropic context windows guide.
The context window also includes system prompts, tool definitions, images, and documents, all counting toward the limit. Output tokens, including those used for thinking, also count. This makes token counting essential for estimating request sizes before sending them.
Position encoding and context length
Transformers use position encoding to incorporate information about token order. Rotary Position Embedding (RoPE) encodes absolute position with a rotation matrix while also capturing relative position dependencies in self-attention. This method allows flexibility in sequence length and shows decaying inter-token dependency with increasing relative distances, as presented in the RoFormer paper.
Position encoding is crucial for models to understand the order of tokens, which affects how they use context. Different encoding schemes can influence how well a model handles long sequences. RoPE is one approach that has been integrated into models like RoFormer and is available in Hugging Face.
The choice of position encoding can affect the maximum context length a model can handle and how effectively it uses that context. Understanding these mechanisms helps in evaluating model specifications and limitations.
Practical considerations
When working with LLMs, it is important to check the context window size specified for each model. For example, some models offer up to 1 million tokens, while others have 200,000 tokens, as noted in the Anthropic model comparison. The maximum output length is also limited separately.
Inputs can include images and PDFs, with limits on the number of pages or images per request. For instance, a single request can include up to 600 images or PDF pages for models with a 1M-token window, but fewer for smaller windows. Reaching request size limits may occur before token limits.
To estimate token usage before sending a request, developers can use token counting APIs. This helps avoid exceeding the context window and ensures efficient use of the model's capacity. Understanding the trade-offs between context size, cost, and performance is essential for building effective AI applications.
| Source | Definition | Token measurement | Performance note |
|---|---|---|---|
| IBM | Amount of text in tokens the model can consider | Tokens, ~1.5 tokens per word | Larger windows increase accuracy but also cost |
| Anthropic | All text the model can reference, including response | Tokens, with token counting API | Accuracy degrades with more context (context rot) |
| Context is surrounding information for prediction | Tokens, ~4 characters per token in English | N-grams offer limited context | |
| AWS | Not explicitly defined, but inputs up to 100K tokens for some models | Tokens | Transformers process sequences in parallel |
| arXiv Lost in the Middle | Input context for question answering | Not specified | Performance degrades when info is in the middle |
| arXiv RoFormer | Sequence length flexibility via position encoding | Not specified | RoPE enables flexibility and decaying dependency |
Frequently asked questions
What is a context window in a language model?
A context window is the amount of text, measured in tokens, that a language model can consider when generating a response. It acts as the model's working memory, limiting how much input it can process at once.
How are tokens counted in a context window?
Tokens are the smallest units of text, which can be characters, subwords, or whole words. The number of tokens per word varies by language and tokenizer; for English, roughly 1.5 tokens per word is a common estimate.
Does a larger context window always improve performance?
No, larger context windows do not guarantee better performance. Accuracy can degrade as token count grows, a phenomenon called context rot, and models may not effectively use information in the middle of long contexts.
What happens when input exceeds the context window?
When input exceeds the context window, it must be truncated or summarized for the model to proceed. This can lead to loss of information and reduced response quality.
How can developers manage context window limits?
Developers can use token counting APIs to estimate request sizes, employ prompt caching, and use compaction strategies for long-running conversations. Curating what goes into the context is also important.