Context Window Explained — LLM Memory Limits

Photo: Je Hwan Lee / Pexels
A context window is the maximum number of tokens a large language model (LLM) can process in a single request, counting both the input prompt and the generated output combined. It is the hard ceiling on how much information the model can "see" at once — there is no persistent memory outside this window unless you add external systems like retrieval-augmented generation (RAG) or vector databases.
If you exceed the context window, the model either refuses the request or the surrounding application silently truncates earlier text. Understanding context windows is essential for building reliable AI applications, controlling costs, and avoiding subtle quality regressions that show up only when conversations or documents get long.
What Is a Token?
LLMs do not read characters or words. They read tokens, which are subword units produced by a tokenizer — most commonly Byte Pair Encoding (BPE) or a variant like SentencePiece. A tokenizer breaks "unbelievable" into pieces such as "un," "believ," and "able" because those fragments show up often enough across training text to earn their own slot in the vocabulary.
Think of a token like a syllable card in a word game. You are not handed whole words to work with, and you are not handed individual letters either — you get chunks that are common enough to be efficient but flexible enough to build almost any word by combining a few of them.
Approximate ratios for English text:
- 1 token ≈ 4 characters
- 1 token ≈ 0.75 words
- 100 tokens ≈ 75 words
- 1,000 tokens ≈ ~750 words or 1.5 pages of single-spaced text
Code, numbers, JSON, and non-English languages (especially Chinese, Japanese, Arabic, and Korean) tokenize less efficiently because BPE vocabularies are built from training corpora that skew heavily toward English web text. A word that takes one token in English might take three or four tokens in Japanese, because the tokenizer never learned a single efficient chunk for it. Practically, this means a 200,000-token context window advertised for English use can shrink to an effective 60,000-80,000 usable tokens once you are writing or pasting large volumes of Japanese or Korean text.
This matters for cost and planning, not just curiosity. If you are building a product for a non-English market, your token budget and your dollar budget are both smaller than the marketing page implies, and you should benchmark real token counts with your actual content before sizing a deployment.
Why Context Windows Have Limits
The Transformer architecture, introduced by Vaswani et al. in "Attention Is All You Need" (2017), uses a mechanism called self-attention, where every token compares itself against every other token in the sequence to decide what to pay attention to. That comparison step is what lets a model connect "it" in sentence 40 back to "the battery" mentioned in sentence 2. But comparing every token to every other token means the amount of work scales as O(n²) with sequence length: double the input, and you roughly quadruple the compute and memory needed for the attention calculation.
A useful analogy: imagine a conference call where every participant has to individually confer with every other participant before the meeting can conclude. A 10-person call has 45 possible pairs to check. A 20-person call — twice as many people — jumps to 190 pairs, more than four times as many. That is the quadratic wall LLMs run into as context grows, and it is why doubling a context window is never just "twice the hardware."
Concretely, three things get harder as sequence length grows:
- GPU memory for the attention matrix (the grid of "how much should token A pay attention to token B" scores) grows quadratically, and that matrix has to sit in expensive high-bandwidth GPU memory during inference.
- Inference latency for the first generated token grows roughly linearly to quadratically depending on the attention implementation, meaning long prompts create a noticeable delay before the model starts responding at all.
- Training cost explodes, because training on long sequences means running that same quadratic attention calculation billions of times across a training run, so every long-context model requires dedicated engineering investment rather than just "turning up a setting."
Modern long-context models get around the worst of this with engineering tricks rather than brute-force scaling. FlashAttention (Dao et al., 2022) restructures the attention calculation to minimize slow memory transfers on the GPU, delivering the same mathematical result faster and with less memory overhead. Sliding window attention limits each token to attending only to a fixed-size local neighborhood instead of the entire sequence, trading some long-range awareness for speed. Ring attention distributes the attention computation across multiple GPUs so no single chip has to hold the full quadratic matrix. Mixture-of-experts (MoE) architectures, used in models like Llama 4 and DeepSeek, route each token through only a subset of the model's total parameters, which reduces the per-token compute cost and indirectly makes room for longer contexts within the same hardware budget.
Evolution of Context Windows (2020-2026)
The frontier has moved from roughly 2,000 tokens to over a million in six years — more than a 500x increase in the largest publicly available windows.
Table 1. Context window growth across major LLM releases, 2020-2026
| Model | Release | Context Window |
|---|---|---|
| GPT-3 | June 2020 | 2,048 tokens (later extended to 4K) |
| GPT-3.5 | November 2022 | 4,096 tokens |
| GPT-4 | March 2023 | 8,192 tokens (later 32K) |
| Claude 1 | May 2023 | 100,000 tokens |
| GPT-4 Turbo | November 2023 | 128,000 tokens |
| Claude 2.1 | November 2023 | 200,000 tokens |
| Gemini 1.5 Pro | February 2024 | 1,000,000 tokens (2M in limited preview) |
| Llama 3.1 | July 2024 | 128,000 tokens |
| Magic.dev LTM-2-Mini | August 2024 | 100,000,000 tokens (research demo) |
| Claude 3.5 Sonnet | 2024-2025 | 200,000 tokens |
| Llama 4 Scout | April 2025 | 10,000,000 tokens |
| Gemini 2.5 Pro | 2025-2026 | 1,000,000 tokens standard tier, 2,000,000 on select enterprise access |
| Claude Sonnet 4.6 / Opus 4.8 | 2026 | 1,000,000 tokens generally available |
| GPT-5.5 / GPT-5.6 | April-June 2026 | ~1,000,000-1,050,000 tokens (API); 400,000 in Codex |
Sources: OpenAI model documentation, Anthropic model comparison, Google DeepMind Gemini 1.5 technical report, Meta AI Llama 4 blog.
A few of these numbers deserve unpacking rather than a glance. Magic.dev's LTM-2-Mini, announced in August 2024, advertised a 100-million-token context — roughly 10 million lines of code, or about 750 novels' worth of text in one prompt. It was a research demonstration of a new attention architecture rather than a broadly shipped commercial product, and it is worth noting as evidence of where the ceiling can go, not as a number you should expect to rent by the API call today.
Llama 4 Scout's 10-million-token figure is real and shipped, but independent analysis after release found that the model's effective usable context — the point at which retrieval accuracy on real tasks starts degrading — sits well below the advertised maximum, a pattern that shows up across nearly every long-context model on the market (more on that in the next section). Meta's own technical documentation notes Scout was pre-trained and post-trained at a 256K context length, with the 10M figure reflecting length-generalization capability layered on top rather than the length the model saw repeatedly during training.
Gemini 2.5 Pro's headline number also needs a footnote. Google ships the model with a 1-million-token window on the standard API, with a 2-million-token tier available to select enterprise customers on Vertex AI rather than universally. If you are pricing out a project against the 2M figure you saw in a blog post, confirm which access tier that number applies to before you commit engineering time to a design that assumes it.
The "Lost in the Middle" Problem
Large context windows do not guarantee uniform attention across all that text. Liu et al. (Stanford University, July 2023) demonstrated in "Lost in the Middle: How Language Models Use Long Contexts" that LLMs reliably recall information placed at the beginning and end of a long context but show measurably worse recall for facts buried in the middle — a U-shaped performance curve that held across every model the researchers tested at the time.
Picture a hiring manager who skims a 40-page resume packet. She reads the first page carefully, remembers the closing page because it's fresh, and skims through the middle thirty pages fast enough that details there barely register. That is roughly what happens inside a Transformer processing a long prompt: positional and attention patterns favor the edges of the sequence over its center, even though the architecture is technically capable of attending to every token.
Practical implications for anyone building with long-context models:
- Place the most important instructions and data at the start or end of the prompt, not buried in a large pasted document in between.
- Do not assume a model will reliably find a single fact sitting at the 50% mark of a 200,000-token context, even if the model's marketing claims "100% recall."
- Use the "needle-in-a-haystack" benchmark — inserting a specific, verifiable fact into a long block of unrelated text and asking the model to retrieve it — as a first-pass sanity check on any long-context deployment, but do not stop there.
Do newer models still show the lost-in-the-middle effect?
Partially. Frontier models released since 2024 — the Claude 3 family, Gemini 1.5 and 2.5, and GPT-4 Turbo and later — score above 95% on single-needle needle-in-a-haystack tests across their full advertised context length. But single-needle retrieval is the easiest version of the problem, and it is not representative of real work.
What is the RULER benchmark and why does it matter more than needle-in-a-haystack?
RULER, introduced by NVIDIA researchers in April 2024, is a synthetic benchmark that goes beyond single-fact retrieval to test multi-hop tracing (following a chain of references across the document), aggregation (counting or summarizing scattered information), and multi-needle retrieval (finding several facts at once, not just one). The RULER paper's central finding is stark: models that score near-perfect on vanilla needle-in-a-haystack fail to hold that performance on RULER's harder task types as input length grows, and the researchers found that even when a model claims a context size of 32,000 tokens or more, only about half of tested models actually maintain acceptable performance at that length. This is the difference between a model's advertised context window and its effective context window, and the gap between the two is often the single most important variable in whether a long-context application works in production. (RULER paper, arXiv 2404.06654)
That gap is why serious AI teams do not take a vendor's context-window number at face value. They run their own retrieval and reasoning evaluations against representative documents before shipping a feature that depends on long-context recall.
What to Do When Content Exceeds Context
Even with million-token windows now common, plenty of real workloads exceed them — a legal discovery process with tens of thousands of documents, a codebase with millions of lines, or a customer support history spanning years. When content exceeds the window, or when you want to avoid relying on lost-in-the-middle-prone recall even inside the window, five strategies cover most production cases.
Table 2. Strategies for handling content that exceeds (or strains) the context window
| Strategy | What it does | Best use case |
|---|---|---|
| Chunking | Splits long documents into smaller independent pieces, processes each, then merges results | Batch summarization of many similar documents |
| RAG (retrieval) | Indexes content in a vector database, retrieves only the relevant chunks per query at inference time | Q&A over large, frequently updated knowledge bases |
| Summarization | Compresses earlier conversation turns or document sections into a running summary | Long chatbot conversations, meeting transcripts |
| Structured extraction / distillation | Pre-extracts key facts into structured data (JSON) once, then reuses the compact extraction instead of the raw source | Repeated querying of the same static document set |
| Hierarchical map-reduce | Summarizes each chunk individually, then summarizes the summaries | Book-length documents, legal contracts, research corpora |
Of these, RAG has become the default choice for most production applications precisely because it sidesteps both the token-cost problem and the lost-in-the-middle problem: instead of stuffing an entire knowledge base into the prompt and hoping the model finds the right paragraph, you retrieve only the handful of passages that are actually relevant to the current question and hand the model a short, targeted context where every included fact matters. See our guide to retrieval-augmented generation (RAG) for the underlying mechanics, including how embeddings power the retrieval step.
Should I just use a bigger context window instead of building RAG?
Not by default. A 2026 industry pattern that shows up repeatedly in engineering postmortems: teams that dump entire knowledge bases into a million-token prompt see higher latency, higher cost per query, and — because of lost-in-the-middle effects — worse accuracy than a well-tuned RAG pipeline retrieving a few thousand relevant tokens. Bigger windows are a tool for genuinely context-dependent tasks (reviewing an entire codebase's cross-file dependencies, for instance) rather than a universal replacement for retrieval.
Cost Scaling
Longer context costs more money, and the relationship is not always linear. Providers bill by tokens processed, typically priced per million input tokens, and several providers apply a higher rate above a size threshold to account for the added compute burden of longer sequences.
Illustrative 2026 pricing patterns (check current provider pricing pages before budgeting, as these change frequently):
- Anthropic Claude Sonnet tier: a base per-million-token rate below 200K context, with a documented premium rate applied above that threshold.
- OpenAI GPT-4o-class models: priced per million input tokens with output tokens billed separately at a higher rate, consistent with the industry-standard input/output split.
- Google Gemini 1.5 Pro: historically priced lower per token below 128K context and roughly double that rate above it, illustrating how the quadratic compute cost of attention translates directly into tiered pricing.
The arithmetic adds up quickly in agentic workflows. If an AI agent re-sends the same long conversation history or document context with every tool call in a multi-step loop — which is the default behavior in many naive agent implementations — a session that makes twenty calls against a 200,000-token context can rack up a meaningfully larger bill than a single one-shot query would suggest, simply because the same tokens are being re-billed on every round trip.
Prompt Caching — The Cost Antidote
Anthropic launched prompt caching in August 2024, with OpenAI and Google introducing comparable features afterward. The idea is straightforward: if the beginning portion of your prompt — a system prompt, a large reference document, a codebase snapshot — stays identical across multiple requests, the provider can cache the processed representation of that prefix instead of recomputing it from scratch every time.
Anthropic's implementation bills cached prefix tokens at roughly 10% of the normal input cost, with latency reductions reported as high as 85% for cache hits. OpenAI's automatic prompt caching applies a discount closer to 50% off the standard input rate for repeated prefixes. The exact percentages vary by provider and change over time, so treat any specific number as a snapshot rather than a permanent guarantee, and confirm current rates against the provider's live pricing documentation before finalizing a cost model.
This makes large-context workflows economically viable in a way flat per-token pricing does not. A team indexing a 150,000-token codebase for a coding assistant can process that codebase once, cache it, and have every subsequent query in the session reuse the cached prefix at a fraction of the original cost, rather than paying full price to re-read the entire codebase on every single question. Anthropic's default cache time-to-live is five minutes, refreshed on each cache hit, with a longer one-hour cache option available at a higher write-side cost for workloads with sparser request patterns. Google's Gemini implements explicit context caching billed hourly per cached token, which suits workloads with predictable, sustained reuse rather than bursty conversational patterns.
Read our deep dive on prompt caching for LLMs for implementation patterns and provider-by-provider setup details.
How much can prompt caching actually save on a real workload?
For workloads with a large, stable prefix reused across many calls — a coding assistant re-reading the same repository, or a customer support bot re-reading the same product documentation — caching can cut the effective cost of that prefix by roughly 80-90% after the first request, since only the first call pays full price to populate the cache while every subsequent call within the cache's time-to-live window pays the discounted cached rate.
Context Window vs. Training Data vs. Memory
These three concepts get conflated constantly, and mixing them up leads to real product bugs — like assuming a chatbot "remembers" a user's preference from last week when it has no mechanism to do so.
Table 3. Three distinct concepts that are easy to confuse
| Concept | What it actually is | How it changes |
|---|---|---|
| Training data | The text corpus the model learned statistical patterns from during training | Frozen after training completes; updating it requires retraining or fine-tuning |
| Context window | The tokens the model can see in this single request, right now | Resets completely between separate API calls unless the application resends prior turns |
| Memory | Persistence of information across separate conversations or sessions | Does not exist natively in the model; requires an external system such as a database, vector store, or memory-management layer |
The model does not remember anything between requests unless the surrounding application explicitly re-sends the prior conversation turns as part of the new context window, or unless it is connected to an external memory system. One increasingly common way to wire up that external memory and tool access in a standardized way is the Model Context Protocol (MCP), an open standard that lets a model pull in live data from outside systems rather than relying on either the frozen training data or a manually re-sent conversation history.
What People Get Wrong About Context Windows
The single most common misconception is treating the advertised context window as a performance guarantee rather than a hardware ceiling. A model that supports one million tokens is not thereby claiming it will use all one million tokens equally well — as the RULER benchmark data above shows, the gap between "the model will accept this many tokens" and "the model will accurately reason across this many tokens" can be substantial, and it grows with task complexity.
A second common error is assuming context window size and knowledge are the same thing. A model with a 200,000-token context window is not smarter or more knowledgeable than one with a 32,000-token window — the window only governs how much text you can hand it in a single request. Two identically-trained models differing only in context window length will produce identical answers to a short question; the difference only shows up when you need the model to work with large volumes of input text at once.
A third mistake is ignoring output tokens when budgeting a request. Input and output share the same context window on virtually every major provider. If a model has a 128,000-token window and the input consumes 120,000 tokens, only 8,000 tokens remain for the entire response — which can silently truncate a long generated answer or code file with no warning beyond a stop reason in the API response.
The takeaway
A context window is a hardware and architecture constraint, not a design choice a provider makes arbitrarily — it exists because Transformer attention scales quadratically with sequence length, and every trick used to push windows past a few thousand tokens toward today's million-plus figures (FlashAttention, sliding windows, ring attention, MoE routing) is an engineering workaround for that one quadratic wall. The advertised maximum and the effective, reliably-usable maximum are two different numbers, and the gap between them — visible in benchmarks like RULER — is often the deciding factor in whether a long-context feature actually works in production. For most applications, the practical answer is not to chase ever-larger windows but to combine a right-sized model with retrieval, caching, and careful prompt structure that puts the important information where the model reliably looks: the beginning and the end.
FAQs
Q: What is the context window of GPT-5? A: The GPT-5 family evolved through 2026, with GPT-5.4 (March 2026) shipping roughly 1.05 million tokens, GPT-5.5 (April 2026) offering 1 million tokens via API and 400,000 in Codex, and GPT-5.6 (June 2026) holding at roughly 1.05 million tokens with a 128,000-token output cap. Always check OpenAI's current model documentation for the specific variant and access tier you are using, since these numbers shift with each point release.
Q: Is a bigger context window always better? A: No. Bigger windows cost more per request, run slower for the first generated token, and suffer from lost-in-the-middle recall degradation on complex tasks even when they pass simple retrieval benchmarks. For most production applications, RAG paired with a mid-size context model outperforms dumping an entire knowledge base into a million-token window, both on cost and on accuracy.
Q: How do I count tokens before sending a request?
A: Use the provider's official tokenizer library: tiktoken for OpenAI models, the Anthropic SDK's token-counting endpoint for Claude, and the Vertex AI tokenization utilities for Gemini. As a rough estimate without any tooling, assume 1 token equals about 0.75 English words, but treat this as a planning heuristic, not a substitute for exact counts on cost-sensitive workloads.
Q: What happens when I exceed the context window? A: Direct API calls typically return an error, often an HTTP 400 response with a token-limit message identifying how many tokens were sent versus the model's maximum. Some consumer chat interfaces instead silently truncate the oldest messages in a conversation to stay under the limit, which can cause the model to behave as if it has forgotten earlier context with no visible warning to the user.
Q: Does the context window include the model's output, or just the input? A: On essentially every major provider, input and output tokens share the same total context window budget. If a model has a 128,000-token window and your input consumes 120,000 tokens, only 8,000 tokens remain for the entire generated response, which can truncate a long answer partway through.
Q: Why does my code or non-English text take more tokens than my English prose? A: BPE tokenizers are trained on corpora that skew heavily toward English web text, so common English word fragments get efficient single-token representations while code syntax, JSON structure, and non-Latin scripts like Japanese or Arabic often require two to four times as many tokens to represent the same visible content, since the tokenizer never learned compact chunks for those patterns.
Q: What is the difference between a model's advertised context window and its effective context window? A: The advertised window is the maximum number of tokens the model architecture will technically accept without erroring out. The effective window, as measured by benchmarks like RULER, is the length at which the model still performs accurately on realistic tasks such as multi-hop reasoning or multi-fact retrieval. RULER's research found that even models claiming 32,000 tokens or more of context, only about half maintained acceptable performance at that stated length, meaning the two numbers diverge more often than marketing materials suggest.
Q: Can I trust a model's needle-in-a-haystack score as proof it handles long documents well? A: Treat it as a floor, not a ceiling. Needle-in-a-haystack tests a single, isolated fact retrieval, which is the easiest version of long-context reasoning. Harder benchmarks like RULER that test multi-hop tracing and multi-needle retrieval show meaningfully lower scores on the same models, so a perfect needle-in-a-haystack result does not guarantee the model will correctly synthesize information scattered across a long real-world document.
Q: Does prompt caching reduce the size of the context window, or just its cost? A: Only cost and latency, not size. Prompt caching stores the processed representation of a repeated prompt prefix so the provider does not have to recompute it on every call, cutting the price of reusing that prefix by roughly 80-90% in Anthropic's implementation and about 50% in OpenAI's. It does not increase how many tokens the model can process in total.
Q: Should a startup build with RAG or just pay for the largest available context window? A: For most applications with a knowledge base larger than a few hundred pages or one that updates regularly, RAG is the more reliable and cost-effective choice, because it avoids both the linear cost growth of re-sending large contexts and the lost-in-the-middle recall problem. Large context windows are better suited to tasks that genuinely require holistic understanding of a single large document, such as reviewing cross-file dependencies in one codebase snapshot, rather than serving as a general substitute for a retrieval system.
Sources
- Vaswani et al., "Attention Is All You Need" (2017) — original Transformer architecture paper
- Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (Stanford University, July 2023)
- Hsieh et al., "RULER: What's the Real Context Size of Your Long-Context Language Models?" (NVIDIA, April 2024)
- Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (2022)
- Google DeepMind, "Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context" (February 2024)
- Meta AI, "The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation" (April 2025)
- Anthropic, prompt caching and model context documentation (docs.anthropic.com)
- OpenAI, model and tokenizer documentation (platform.openai.com, tiktoken)
Related Articles
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles

Embeddings Explained — How Vector Search Works
Embeddings turn text and images into vectors that capture meaning. Learn how vector search, RAG, and similarity work — with 2026 models compared.

Test-Time Compute Explained — LLM Reasoning
Test-time compute lets LLMs think longer at inference. Learn how o1, o3, DeepSeek-R1, and Claude reasoning models work and when to use them.

How to Write AI Prompts That Actually Work
The quality of an AI model's output is determined largely by the quality of the input it receives. A prompt is not a search query — it is the only communication channel between the user and the model. Every word specifies or leaves ambiguous something about the desired output. This guide covers the principles behind effective prompting, the techniques with the most research evidence behind them, and the common mistakes that produce frustrating, generic results.

Why AI Models Give Different Answers to the Same Question
Send ChatGPT the same message twice within five minutes and you will likely receive two distinct responses. The wording differs, the examples change, sometimes the conclusions shift. This is not a bug, and it is not the model lying — it is the intended consequence of probabilistic sampling, a deliberate design choice that trades consistency for coverage. Understanding why requires a look at how language models generate text one token at a time.

Advanced Prompt Engineering: Techniques That Actually Work
Beyond 'be clear and specific': the advanced prompt engineering techniques that hold up in published tests, and the popular advice that does not.

What Is RAG? Retrieval-Augmented Generation Explained
RAG is the technique making AI tools actually useful for real work — letting models look things up instead of relying on memory. Here's how it works and why it matters.