Cut LLM API Costs with Prompt Caching: Pricing Mechanics and Practical Patterns

2026-07-11

Prompt caching lets the model provider reuse the unchanged prefix of your prompt across requests and bill those repeated tokens at roughly 10% of the normal input rate — a 90% discount on current Anthropic and OpenAI models. If your application sends a long system prompt, runs agent loops, or injects retrieved documents into every call, caching is usually the single biggest lever for cutting your LLM API bill. This post walks through what actually gets cached, both providers' official cache pricing, how cached tokens are billed on CaMeL Hub, and concrete prompt-structuring rules that turn the discount into real savings.

What prompt caching actually is

Every time an LLM processes your request, it first runs a forward pass over the entire prompt to build internal state (the attention key-value cache) before generating a single output token. For a 50,000-token prompt, that prefill work dominates both latency and cost — and if 48,000 of those tokens are identical to your previous request, the provider is redoing computation it already did seconds ago.

Prompt caching fixes this: the provider keeps the processed state of a prompt prefix for a short window and, when a new request arrives with the exact same prefix, skips recomputing it. Because the work saved is real, both major providers pass most of it back as a price cut — cached tokens are billed at a small fraction of the regular input rate.

The key mechanical detail is that caching is strictly prefix-based. OpenAI's prompt caching guide states it plainly: "Cache hits are only possible for exact prefix matches within a prompt." One changed character at position 100 invalidates everything after position 99. That single fact drives every practical recommendation later in this post.

Where caching pays off: long system prompts, agent loops, RAG

Caching only helps when a large chunk of your prompt repeats across requests within the cache lifetime. Three workload shapes dominate in practice:

• Long system prompts — a customer-support bot with a 20,000-token policy document and few-shot examples resends that block on every user message. With caching, you pay full price once, then ~10% for every subsequent hit.
• Agent loops — coding agents and tool-using agents are the extreme case. Each tool call appends a little context and resends everything before it, so a 40-step run reprocesses the (growing) prefix 40 times. Cache-heavy agent traffic routinely shows 90%+ of input tokens as cache reads.
• RAG and long-document chat — if users ask multiple questions about the same retrieved corpus or uploaded contract, placing the documents at the start of the prompt makes every follow-up question a cache hit on the expensive part.
• Where it does NOT help: one-shot requests with unique content, prompts below the minimum cacheable size (typically 1,024 tokens), or traffic so sparse that the cache expires between calls.

Anthropic's cache pricing: reads at 0.1x, writes at 1.25x

Anthropic's caching is explicit: you mark cache breakpoints in the request with cache_control, and the usage object reports cache writes and reads separately. According to Anthropic's prompt caching documentation, the multipliers relative to the base input-token price are: cache writes with the default 5-minute lifetime cost 1.25x, writes with the optional 1-hour lifetime cost 2x, and cache reads cost 0.1x. Reading a cached prefix also refreshes its lifetime, so a continuously active conversation keeps its cache alive indefinitely.

In dollar terms, per Anthropic's published table (https://claude.com/pricing): Claude Opus 4.8 input is $5 per million tokens, so a 5-minute cache write costs $6.25/M and a cache hit only $0.50/M. The break-even math is trivial — the 25% write premium is repaid by the very first hit, which saves 90%. Any prefix reused at least once within five minutes is worth caching.

One constraint to know: the minimum cacheable prompt length is model-dependent — 1,024 tokens for Claude Opus 4.8 and Claude Sonnet 5 according to the same documentation, with other models ranging from 512 to 4,096. Shorter prefixes are simply processed uncached, with no error and no discount.

import anthropic

client = anthropic.Anthropic(
    api_key="sk-...",  # your CaMeL Hub API key
    base_url="https://api.camel-hub.com",
)

resp = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT,  # policies, tool docs, examples
            "cache_control": {"type": "ephemeral"},  # cache breakpoint
        }
    ],
    messages=[{"role": "user", "content": question}],
)

print(resp.usage)
# cache_creation_input_tokens=...  <- billed at 1.25x input
# cache_read_input_tokens=...      <- billed at 0.1x input

OpenAI's cache pricing: automatic, cached input at 10%

OpenAI takes the opposite design stance: caching is fully automatic. Per its prompt caching guide, "Prompt Caching works automatically for eligible requests, with no code changes required" — any prompt of 1,024 tokens or more is eligible, and hits appear in the response usage under prompt_tokens_details.cached_tokens.

The discount is the same order of magnitude as Anthropic's. According to OpenAI's API pricing page (https://developers.openai.com/api/docs/pricing), cached input is billed at 10% of the regular input rate across the current GPT lineup: gpt-5.5 input is $5.00 per million tokens versus $0.50 cached; gpt-5.4 is $2.50 versus $0.25; gpt-5.4-mini is $0.75 versus $0.075.

Write costs differ by generation. The guide notes that cache writes carry no additional fee on models before the GPT-5.6 family, while on GPT-5.6 and later, writes cost 1.25x the uncached input rate — matching Anthropic's structure. Cache lifetime also improved with GPT-5.6: a cached prefix "remains eligible for reuse for at least 30 minutes," whereas earlier models keep prefixes for roughly 5–10 minutes of inactivity, up to an hour.

{
  "usage": {
    "prompt_tokens": 12083,
    "completion_tokens": 486,
    "prompt_tokens_details": {
      "cached_tokens": 11776
    }
  }
}

How CaMeL Hub bills cached tokens

CaMeL Hub passes the provider's cache economics through to you instead of flattening everything to the full input price. When a request hits the cache, the cache-read portion of your prompt is billed at the provider's cache-read rate — not the regular input rate — and the split is visible per request in your usage logs: cached token counts are recorded alongside normal input and output tokens, and whenever caching saved you money the log shows the original pre-cache price struck through next to what you actually paid.

This works the same whether you call Claude models through the Anthropic-native /v1/messages endpoint with explicit cache_control breakpoints, or call GPT models through the OpenAI-compatible /v1/chat/completions endpoint and let automatic caching do its thing — one API key, one base URL, both caching schemes intact.

How much does this matter in practice? In our production traffic, across cache-heavy Claude workloads (mostly coding agents and long-context assistants), we measured about 30% total cost savings from cache-read billing — that is an internal measurement over real workloads, not a projection. Your number depends entirely on your hit rate: a chat app with short prompts might see single digits, while a busy agent pipeline can do considerably better than 30%.

Structuring prompts to maximize cache hits

Because matching is exact-prefix, cache optimization is mostly about ordering: everything stable goes first, everything volatile goes last. These rules cover the large majority of missed savings we see in real traffic:

• Static content first, in a fixed order: system prompt, then policies and reference documents, then tool definitions and few-shot examples. Never shuffle these between requests.
• Keep volatile values out of the prefix: timestamps, request IDs, session names, or a random user greeting at the top of the system prompt guarantee a 100% miss rate. Put them at the end of the final user message instead.
• Append, don't rewrite: keep conversation history append-only. Editing, summarizing, or reordering earlier turns invalidates the prefix from that point on — if you must compact history, do it rarely and accept one cold request.
• Serialize deterministically: if you build prompts from structured data, fix the JSON key order and whitespace. Two semantically identical payloads with different key order are different prefixes.
• Stay above the minimum: prompts under 1,024 tokens are not cacheable on either provider — no need to pad, but don't expect savings there.
• Keep the cache warm: hits must land within the cache lifetime (5 minutes default on Claude, refreshed by each hit; at least 30 minutes on GPT-5.6+). For bursty batch jobs, grouping requests that share a prefix back-to-back beats spreading them over hours.

from datetime import datetime, timezone

# BAD - volatile data first: the prefix changes on every request,
# so nothing is ever reused
system = (
    f"Current time: {datetime.now(timezone.utc)}\n"
    "You are a support agent.\n"
    f"{POLICY_DOC}"
)

# GOOD - static prefix first, volatile data last
system = f"You are a support agent.\n{POLICY_DOC}"
user = (
    f"{question}\n\n"
    f"(current time: {datetime.now(timezone.utc):%Y-%m-%d %H:%M} UTC)"
)