Prompt Caching

What It Is

Prompt caching avoids recomputing the same prompt prefix on every request. Anthropic offers explicit cache breakpoints (up to 4, 90% savings), OpenAI provides automatic caching (50% savings, no code changes), and Google Gemini supports both implicit and explicit caching. All require a minimum of ~1,024 tokens to activate. OpenAI recently added 24-hour extended retention.

Why It Matters

Prompt caching is Proven because it's become invisible infrastructure. Claude Code's entire architecture depends on it for conversation context. Any production LLM application with system prompts, tool definitions, or multi-turn conversations benefits immediately. One Anthropic customer saved $50,000+ per month on academic paper analysis.

The practical reality: if you're not using prompt caching in production, you're overpaying by 50-90%. It's one of the few optimisations that requires near-zero effort for substantial cost reduction.

Key Developments

• Jan 2026: arXiv paper "Don't Break the Cache" evaluates prompt caching for long-horizon agentic tasks.
• Early 2026: OpenAI adds 24-hour extended caching via prompt_cache_retention parameter.
• Late 2025: Gemini 2.5 enables auto-caching with guaranteed discounts on explicit context caching.
• Ongoing: All three major providers now support caching. Anthropic reports 85% latency reduction on long prompts.

What to Watch

The fragility of prefix ordering is the pain point. Adding or removing a tool, shuffling system prompt sections, or inserting timestamps breaks the cache entirely. Watch for providers to offer more flexible cache invalidation strategies — partial prefix matching or content-addressed caching would be a significant improvement over the current exact-prefix requirement.