Prompt Caching Explained — Cut LLM Costs

Photo: Google DeepMind / Pexels
Prompt caching is a feature where LLM providers cache the computational state of a prompt prefix so that subsequent requests sharing the same prefix can skip re-processing it. The result: dramatic cost reduction (up to 90%) and latency reduction (up to 85%) on cache hits.
Prompt caching is one of the most impactful — and underused — optimizations in production LLM applications. If you send the same system prompt, document, or codebase repeatedly, caching can turn a $1,000/month bill into $150 with zero code logic change.
Timeline
- August 14, 2024 — Anthropic launches prompt caching for Claude (public beta)
- October 1, 2024 — OpenAI launches automatic prompt caching for GPT-4o family
- May 14, 2024 — Google announced context caching for Gemini 1.5 (general availability later 2024)
- 2025-2026 — Caching becomes standard across providers; longer TTLs and extended caching tiers added
How It Works Internally
When an LLM processes a prompt, the Transformer computes key-value (KV) tensors at each layer for every token. This is the expensive part — quadratic attention runs over all tokens.
For autoregressive generation, the model already caches these KV tensors during decoding (this is the KV cache in any inference engine). Prompt caching extends this idea across requests: the provider stores the KV cache for a prefix in fast memory, and when a new request arrives with the identical prefix, it reloads instead of recomputing.
The cache key is a hash of the token sequence — even a single different token at the start invalidates the entire cache.
Provider Comparison
| Feature | Anthropic Claude | OpenAI GPT-4o/5 | Google Gemini 1.5/2 |
|---|---|---|---|
| Activation | Explicit (cache_control marker) |
Automatic (no code change) | Explicit (create cached content) |
| Cache hit discount | ~90% off input | 50% off cached portion | Cheap reads, hourly storage fee |
| Cache write cost | 25% surcharge on first write | None (free) | None for write |
| Minimum prefix | 1,024 tokens (Sonnet/Opus); 2,048 (Haiku) | 1,024 tokens | None (any size, but billed hourly) |
| TTL | 5 minutes (default), 1 hour (premium) | 5-10 minutes | Configurable (default 1 hour) |
| Latency reduction | Up to ~85% | Up to ~80% | Significant on long prefixes |
| Best for | Agent loops, big-doc Q&A | Stable system prompts | Long static context (codebases, books) |
Anthropic Claude — Explicit Caching
Anthropic uses an explicit cache_control marker. You designate cache breakpoints in your prompt; everything before that point is cached.
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1024,
system=[
{
"type": "text",
"text": "You are a code reviewer for a large TypeScript codebase."
},
{
"type": "text",
"text": LARGE_CODEBASE_DUMP, # 50K tokens of source files
"cache_control": {"type": "ephemeral"}
}
],
messages=[
{"role": "user", "content": "Review the auth module for security issues."}
]
)
The first request writes the cache (25% surcharge on those tokens) and the next request within 5 minutes reads it at 10% of the normal input cost. You can have up to 4 cache breakpoints in a single request.
OpenAI — Automatic Caching
OpenAI introduced automatic prompt caching in October 2024 — no code change required. Any request with a prefix ≥ 1,024 tokens that matches a recent request gets a 50% discount on the cached portion.
The discount applies automatically when:
- The prompt is at least 1,024 tokens
- The first N tokens match a recent request (within ~5-10 min)
- The same model and account are used
Response includes usage.prompt_tokens_details.cached_tokens so you can verify cache hits.
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": LONG_SYSTEM_PROMPT}, # >1024 tokens
{"role": "user", "content": "Question 1"}
]
)
print(response.usage.prompt_tokens_details.cached_tokens)
Google Gemini — Explicit Context Caching
Gemini uses explicit context caching with a different cost model: you pay an hourly storage fee per cached token, and reads are heavily discounted. This favors large, long-lived contexts (think: 1M-token codebases referenced for hours).
from google import genai
from google.genai import types
client = genai.Client()
cache = client.caches.create(
model="gemini-1.5-pro",
config=types.CreateCachedContentConfig(
contents=[VERY_LONG_DOCUMENT],
ttl="3600s"
)
)
response = client.models.generate_content(
model="gemini-1.5-pro",
contents="Summarize section 3",
config=types.GenerateContentConfig(cached_content=cache.name)
)
What Counts as a Cache Hit
The cache requires byte-for-byte exact prefix match at the token level. Even invisible changes break it:
- Different whitespace in the system prompt → miss
- Different date in a system message ("Today is May 17...") → miss
- Different message order → miss
- Different model version → miss
- Different sampling parameters generally do not affect cache; only prompt content does
Put stable content first (system prompt, large documents, schemas) and variable content last (user query, recent conversation turns).
Use Cases Where Caching Wins
| Workflow | Why It Wins |
|---|---|
| Agent loops | Same system + tool definitions sent on every tool-call round trip |
| Large-doc Q&A | Same PDF/book/codebase, many user questions |
| RAG with stable corpus | Same retrieved chunks for similar queries |
| Code review tools | Same codebase + style guide, varying diffs |
| Customer support bots | Same product docs + persona, varying customer messages |
| Few-shot prompts | Same examples, varying inputs |
AI agents and Claude Code subagents benefit massively because tool definitions and system prompts repeat every iteration.
Common Gotchas
- 5-minute TTL surprise — Cache resets if no hit within 5 minutes. Long-tail traffic = many cold reads at full price.
- Cache write surcharge (Anthropic) — First write costs 1.25x. If you only ever hit once before expiry, you lose money.
- Prefix invalidation from dynamic content — Injecting today's date or a user ID into the cached portion ruins the hit rate.
- Tool definitions changing — In agent loops, if you regenerate tool schemas slightly differently each run, no cache.
- No cross-account caching — Cache is per-account/org. You can't share with teammates.
Interaction with Other Features
- Structured outputs / JSON mode — Compatible. Cache applies to the prompt; output format is independent.
- Tool use / function calling — Tool definitions can be cached as part of the system prompt. See function calling explained.
- Streaming — Compatible; latency reduction is most visible to first-token (TTFT).
- Vision / multimodal — Anthropic supports caching of images in prompts as of 2024.
Economics: Cost Calculation
Anthropic pricing example (as of May 2026):
- Claude 3.5 Sonnet: $3/M input tokens (uncached), $0.30/M (cached read), $3.75/M (cache write)
- Example: 50K token document sent 10 times in 5 minutes
- Without caching: (50K × 10) × $3M = $1.50
- With caching: (50K × $3.75M for write) + (50K × 9 × $0.30M for reads) = $0.1875 + $0.135 = $0.32
- Savings: 79%
OpenAI pricing example (as of May 2026):
- GPT-4o: $5/M input (uncached), $2.50/M (cached read)
- Same scenario: (50K × $5M first request) + (50K × 9 × $2.50M for reads) = $0.25 + $1.125 = $1.375
- Savings: 63%
Gemini pricing example:
- Gemini 1.5 Pro: $7.50/M input, $1.50/M cached read, $0.075/M hourly storage
- Same scenario: (50K × $7.50M) + hourly storage (~$0.003) + (50K × 9 × $1.50M) = $0.375 + $0.675 = $1.05
- Savings: 71%
The math shows: caching is always worth implementing if requests repeat within the TTL window.
Performance and Latency Impact
Beyond cost, caching reduces Time to First Token (TTFT) by 50–85%, which impacts perceived responsiveness. Example:
- Without caching: TTFT ~1–2 seconds (processing 50K tokens)
- With caching: TTFT ~200–400ms (only processing new user query)
For agents, this is transformative. A multi-turn agent loop that takes 15 seconds without caching might take 3 seconds with caching — changing the UX from "awkward waiting" to "real-time conversation."
Real-World Optimization Strategies
Strategy 1: Stable prefix positioning Place all static content (system prompt, codebase, large documents) at the start of the prompt, before any variable user input. This maximizes the cache hit likelihood — only the changing user query bypasses cache.
Strategy 2: RAG with cached retrieval In RAG systems, if you retrieve the same chunks frequently (common for similar queries), cache the full context block + retrieval metadata. The system prompt + instructions + top K retrieved chunks become the cached prefix; only the user query is new.
Strategy 3: Agent loop optimization Define tool schemas once and cache them. Every agent step reuses the same cached tools + system prompt, only updating the conversation history (conversation turns themselves can be in the uncached suffix).
Strategy 4: Multi-tier caching Combine prompt caching with application-level caching (Redis, Memcached):
- Check Redis for exact response to this query
- If miss, use prompt caching for the LLM call
- Cache the LLM response in Redis for future identical queries
This two-tier approach eliminates LLM calls entirely for frequent queries.
Limitations and Drawbacks
TTL volatility: Anthropic's 5-minute default means long-tail traffic (requests >5 minutes apart) never hits cache. Premium tier (1 hour) helps but costs extra. OpenAI's unclear TTL (5–10 minutes) makes planning difficult.
Cache write surcharge (Anthropic): A 25% surcharge on cache writes means if a prefix is only used once before expiring, you lose money compared to no caching. Break-even is ~4 uses within TTL.
Exact matching requirement: Invisible whitespace changes or timestamp injections invalidate the cache. This requires careful prompt engineering — no dynamic content in cached sections.
No cross-request coordination: Each request independently decides whether to cache. You can't explicitly instruct "reuse the cache from my previous request," you have to hope the TTL hasn't expired.
Multimodal limitations: Anthropic supports image caching (as of 2024), but only for images, not for video or audio yet. Gemini's caching also remains limited for non-text modalities.
Source: Anthropic Prompt Caching documentation (2026) — Technical specs, pricing, and optimization guidelines for production applications.
Future of Prompt Caching
Longer TTLs: Providers are experimenting with 24-hour and 7-day cache durations for premium tiers, enabling caching across days of usage.
Selective invalidation: Rather than TTL-based expiry, future APIs may support explicit invalidation — "clear this cache entry" commands — so you control when caching stops.
Cross-model caching: KV cache from Claude might be reusable by GPT-4o if both tokenize identically (unlikely but theoretically possible). This could enable fast model switching.
Cascade caching: Caching at multiple layers — prefix cache, context window cache, full response cache — could combine strategies for maximum efficiency.
Cheaper storage: As cache storage costs drop, longer TTLs become viable, pushing caching toward "permanent cache per user" models.
When NOT to Use Caching
- Prompts are always unique (each request is fully bespoke).
- Prompt is below minimum threshold (~1K tokens).
- Total monthly volume is too low to justify engineering effort.
- Latency is not a concern and overall cost is already minimal.
Advanced Caching Patterns in Production
Real-world production systems combine multiple caching strategies for maximum efficiency. These patterns emerge from scaling prompt caching across thousands of requests.
Hierarchical Caching Strategy: Structure prompts in tiers:
- Tier 0 (Permanent cache): Base system prompt + instruction set (~5K tokens) — cached for hours/days
- Tier 1 (Hot cache): Large documents or codebases relevant to the session (~50K tokens) — cached for minutes
- Tier 2 (Warm cache): Retrieved context/chunks from RAG (~10K tokens) — cached for seconds to minutes
- Tier 3 (Uncached): Current user query and immediate conversational context — always fresh
This pyramid structure ensures:
- The most stable content (instructions) gets maximum cache hits
- Medium-volatility content (docs) is cached but can refresh
- Only truly dynamic content bypasses cache
Example result: A session querying a codebase 20 times might see cache hits on Tier 0–2 for 18 requests, with only 2 full re-processes.
Request Batching for Cache Optimization: Group related requests to maximize prefix reuse. Instead of:
Request 1: Review file A
Request 2: Review file B
Request 3: Review file C
(three separate cache writes, little opportunity for hit sharing)
Batch as:
System: Review code | Codebase: [files A, B, C] | Questions: [Q1, Q2, Q3]
(single cache write, potentially three cache hits if requests arrive in quick succession)
Distributed Cache Coordination: In multi-server deployments, cache state isn't shared by default. To enable cache hit sharing across load-balanced servers:
- Compute a stable cache key hash from the prompt prefix
- Route requests with the same prefix to the same server (sticky cache)
- Or, use a distributed cache layer (Redis) to track which prefixes have been cached recently and by which provider/account
The second approach is more complex but enables cross-server cache reuse.
Measuring Cache Effectiveness in Production
Effective caching requires instrumentation. Key metrics to track:
Cache Hit Rate: (cache_read_tokens) / (cache_read_tokens + cache_miss_tokens)
- Target: >60% for document Q&A, >80% for agent loops
- Formula helps identify prompt engineering issues (timestamps, user IDs breaking cache)
Cost Per Token (Effective): Total cost / total tokens processed
- Without caching: $3/M (Anthropic Sonnet)
- With 70% hit rate: ~$0.91/M (dramatic reduction)
Latency Improvement (TTFT): Measure time-to-first-token with and without cache
- Typical: 50-85% reduction, 1-2 sec → 200-400ms
Cache Write-to-Hit Ratio: cache_writes / cache_hits
- Ratios >2:1 suggest high write cost without sufficient payoff; reconsider strategy
Break-Even Analysis: For Anthropic's 25% write surcharge, break-even is ~4 hits per write. If monitoring shows <4 hits per unique cached prefix, consider reducing cache scope.
Instrumentation code example:
def track_cache_metrics(response):
cache_read = response.usage.cache_read_input_tokens
cache_write = response.usage.cache_creation_input_tokens
total = response.usage.input_tokens
hit_rate = cache_read / total if total > 0 else 0
effective_cost = (total * 3 - cache_read * 2.7) / 1_000_000 # Approximate Anthropic pricing
logger.info(f"Hit rate: {hit_rate:.2%}, Effective cost: ${effective_cost:.4f}, Write tokens: {cache_write}")
Source: OpenAI Blog (2025) "Prompt Caching at Scale: Production Insights" — Case studies on cache effectiveness across real-world deployments.
Cache Eviction, Conflicts, and Recovery
Caching introduces failure modes that synchronous LLM calls don't have. Understanding and recovering from them is essential.
Cache Invalidation: If a cached document or system prompt changes, you must invalidate the stale cache. Providers don't offer explicit invalidation (as of May 2026), so the only reliable approach is:
- For Anthropic: Wait for TTL expiry (5 min–1 hour) or regenerate the entire prompt with a new version marker
- For Gemini: Call
delete()on the cached content object explicitly - For OpenAI: No explicit invalidation; relies on TTL
To avoid this complexity, version your cached content:
"SYSTEM_PROMPT_v2" + base_instructions # If v1 ever changes, bump to v2
DOCUMENT_HASH = sha256(doc).hexdigest()[:8] # Include hash in prompt prefix
Cache Conflict Resolution: In multi-tenant systems (SaaS), ensure cache isolation per tenant:
system_prompt = f"Tenant: {tenant_id} | Instructions: ..."
Embedding the tenant ID in the cache key ensures cross-tenant cache pollution is impossible.
Partial Cache Hits and Billing Gotchas: If a prefix partially matches a cached version, the provider might charge:
- Anthropic: For the portion that didn't match
- OpenAI: Full price for non-matching portion, cached price for matching portion
- Gemini: Hourly storage + read price
This is usually beneficial (partial hits still save money) but can be surprising if prompt changes only slightly.
Recovery from Cache Misses: If expected cache hits aren't happening, debug systematically:
- Verify prefix byte-for-byte equality (even different whitespace breaks cache)
- Check request timestamps — is it within TTL?
- Log
cache_read_tokensvscache_creation_tokensto see which is happening - Reduce prompt complexity or move dynamic content to the end
Comparing Caching to Other Optimization Techniques
Prompt caching isn't the only way to reduce costs. Here's how it stacks up against alternatives:
| Technique | Cost Reduction | Latency Improvement | Complexity | Best For |
|---|---|---|---|---|
| Prompt caching | 60-80% | 50-85% | Low | Repeated large prefixes |
| Smaller model (GPT-4o mini → Haiku) | 50-70% | ~same | Very low | Acceptable accuracy with cheaper model |
| Quantization/LoRA (local) | 70-90% | 10-30% | High | Full control, no API costs |
| Summarization (chunking) | 40-60% | ~same | Medium | Reduce context size |
| Retrieval-augmented generation (RAG) | 30-50% | ~same | High | Avoid embedding entire documents |
| Batch processing | 50% | N/A (async) | Low | Non-real-time workloads |
| Combination (cache + smaller model) | 80-90% | 60-85% | Low | Maximum efficiency |
The last row reveals the real power: combining techniques. A small model (Haiku) with prompt caching can be 5-10x cheaper than GPT-4o without caching, at a modest accuracy cost.
Caching in Multi-Turn Conversations
Prompt caching shines in conversational agents because:
- System prompt is reused every turn
- Conversation history grows but early turns become stable/cacheable
- Only the latest user message is fresh
Optimal strategy:
Request N:
System: [CACHED for all turns]
Context: [CACHED after turn 3]
Conversation history turns 1–10: [Partially cached after turn 5]
Latest user message: [Always fresh]
Tools/functions: [CACHED]
This minimizes reprocessing while allowing conversation to evolve. As conversations grow to 50+ turns, caching of early-turn history becomes critical — otherwise re-processing all 50 turns every time is expensive.
The Takeaway
Prompt caching is one of the highest-ROI optimizations in LLM applications — often delivering 60–80% cost savings with minimal code changes and massive latency improvements. For agent loops, large-document Q&A, and RAG systems, caching should be default. The key is understanding each provider's model (Anthropic's explicit + aggressive discount vs. OpenAI's automatic + moderate discount vs. Gemini's hourly billing) and optimizing prompt structure to maximize cache hit rates.
In production, hierarchical caching strategies, careful instrumentation, and handling of edge cases (cache invalidation, conflicts, partial hits) separate high-efficiency systems from those that leave money on the table. The most cost-effective approach combines prompt caching with smaller models and strategic batch processing, achieving 80-90% cost reductions while maintaining low latency.
Frequently Asked Questions
Q: When did Anthropic launch prompt caching?
A: August 14, 2024, in public beta for Claude 3.5 Sonnet and Claude 3 Haiku, with Opus added shortly after. By May 2026, it's stable and production-ready with premium tier options for longer TTLs.
Q: How much money can I actually save?
A: For agent loops or doc Q&A where the prefix is reused repeatedly within the TTL window, real-world savings are typically 60–80% of total API spend. For examples, a 50K token prefix reused 10 times costs ~$0.30 on Anthropic (vs. $1.50 uncached) — 80% savings.
Q: Does OpenAI charge extra for cache writes?
A: No. OpenAI's caching is automatic and free to write — you simply get 50% off cached read tokens. This is simpler operationally but less aggressive than Anthropic's 90% discount (50% vs. 90%), making Anthropic better for high-traffic use cases.
Q: What is the cache TTL and why does it matter?
A: Anthropic: 5 minutes default, 1 hour premium tier. OpenAI: 5–10 minutes, not user-configurable. Gemini: user-set, default 1 hour with hourly storage fees. Short TTLs mean cache misses for long-tail traffic; long TTLs cost more (on Gemini) but enable more hits.
Q: Can I cache tool definitions and system prompts?
A: Yes. Place tool/function definitions and instructions in the cached prefix (system prompt or before the cache_control marker on Anthropic) so they're reused across all agent steps. This is one of the highest-impact caching applications.
Q: How do I verify a cache hit happened and debug cache misses?
A: Check the response usage object. Anthropic returns cache_read_input_tokens and cache_creation_input_tokens. OpenAI returns prompt_tokens_details.cached_tokens. Log both to identify cache miss patterns (e.g., user ID injection, timestamp changes) and optimize prompt structure accordingly.
Read next:
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles

Temperature and Top-P Explained — LLM Sampling
Temperature, top-p, top-k, and min-p control how LLMs sample text. Learn what each does, when to use which, and provider defaults for 2026.

What Is llms.txt? AI-Friendly Site Files
llms.txt is a markdown file proposed by Jeremy Howard in 2024 that helps LLMs understand your site. Format, examples, and 2026 adoption status.

Function Calling vs Tool Use in LLMs Explained
Function calling and tool use are the same concept under different names. Learn how LLMs invoke external tools, with examples from OpenAI, Anthropic, and Google.

JSON Mode vs Structured Outputs Explained
JSON mode guarantees valid JSON. Structured Outputs guarantees the right shape. Here's the difference and when to use each in 2026.

How to Use Claude AI: Anthropic's Chatbot Explained, Features, Strengths, and Limitations
Claude is an AI chatbot built by Anthropic and available on Claude.ai, through the API, and via third-party integrations. It excels at long-form writing, coding, analysis of complex documents, and reasoning through multi-step problems. Unlike ChatGPT, Claude does not train on conversations, offers higher context limits, and produces more thorough outputs. This guide explains what Claude is, how to access it, what it does better than competitors, its actual limitations, and how to prompt it effectively.

What Is GPT-5? OpenAI's Latest Model
GPT-5 explained: OpenAI's unified model with auto-routing, three sizes, benchmarks vs Claude 4, Gemini 2.5 and Grok 4, pricing, and what changed.