Prompt caching

Prompt caching lets you reuse a large, stable prompt prefix (system instructions, tool definitions, a long document) across many requests. The cached portion is billed at a steep discount on re-use, and on Anthropic models it does not count toward your input-token-per-minute (ITPM) rate limit — so caching is also the cheapest way to raise your effective throughput.

tokenroute is a passthrough gateway: you decide what to cache by putting cache_control in your request, and we forward it to the provider unchanged. The response's usage carries the cache token counts back, and your bill applies the reduced cache-read rate automatically — you don't configure anything on our side.

Which models need cache_control?

So in practice: cache_control only matters for Anthropic (claude-*) models. For OpenAI and DeepSeek, send identical prefixes and the provider caches them for you.

Adding cache_control (Anthropic)

The /v1/chat/completions endpoint is OpenAI-compatible, so you express a cache breakpoint by making a message's content an array of blocks and attaching cache_control to the last block of the stable prefix. tokenroute (via LiteLLM) translates that into Anthropic's native cache control.

Cache the system prompt:

{
  "model": "claude-sonnet-4-6",
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "<large, stable system prompt — instructions, style guide, schema…>",
          "cache_control": { "type": "ephemeral" }
        }
      ]
    },
    { "role": "user", "content": "Summarize the attached ticket." }
  ]
}

Cache a long document that many questions reuse — put the breakpoint at the end of the shared part, leave the varying question uncached:

{
  "model": "claude-sonnet-4-6",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "<50KB contract you'll ask many questions about>",
          "cache_control": { "type": "ephemeral" }
        },
        { "type": "text", "text": "What is the termination clause?" }
      ]
    }
  ]
}

A 1-hour breakpoint (instead of the default ~5 min) is "cache_control": { "type": "ephemeral", "ttl": "1h" }.

Verify it's working

The response usage block reports cache activity. Across repeated requests with an identical prefix you should see reads accumulate:

"usage": {
  "prompt_tokens": 50,                 // uncached remainder (the varying part)
  "cache_creation_input_tokens": 12000, // written to cache this request (~1.25× cost)
  "cache_read_input_tokens": 12000,     // served from cache (~0.1× cost)
  "completion_tokens": 220
}

If cache_read_input_tokens stays 0 across repeated requests, a silent invalidator is breaking the prefix — see the rules below.

Rules that decide whether caching helps

Caching is a prefix match: any byte change anywhere before a breakpoint invalidates the cache from that point on. Get these right or you'll pay the write premium for nothing.

• Stable content first. Put the frozen prefix (system prompt, tool list, document) before the breakpoint; put anything that varies per request (timestamps, UUIDs, the user's question) after it.
• Minimum prefix size. Below ~1,024–4,096 tokens (model-dependent; 4,096 for Opus/Haiku/Sonnet-tier) the breakpoint silently won't cache.
• Max 4 breakpoints per request.
• Don't cache one-shot prompts. A cache write costs ~1.25× a normal read; a cache read costs ~0.1×. You only come out ahead if the prefix is reused. Caching a prefix you'll send once raises your cost.

For AI agents

When wiring tokenroute into a project:

• For claude-* models with a large, stable system prompt or tool definition set that repeats every turn, add one cache_control: { type: "ephemeral" } breakpoint on the last block of that prefix. This is the single biggest lever on both cost and Anthropic rate-limit headroom.
• For gpt-* and deepseek-* models, do nothing — keep the prefix byte-identical and the provider caches it automatically.
• Never add cache_control to a per-request varying block or to a short prompt.

See the API reference for the full request/response surface.