Most conversations about the cost of using a language model are about picking a cheaper model. A lot of the cost, and most of the latency, is in something much less glamorous: how stable the beginning of your prompt is.
Context
Processing a prompt is the expensive step. For every input token, the model computes attention keys and values, and that work grows with the length of the prompt. A long system prompt, a list of tool definitions and a few documents can easily be thousands of tokens, and a naive application recomputes all of it on every call.
Prompt caching removes that repetition. The provider keeps the computed state of a prompt prefix, and when the next request starts with exactly the same tokens, it reuses that state instead of recomputing it. Cached input tokens are billed at a fraction of the normal price, and the first token arrives sooner.
The word that matters is prefix. The match is on the beginning of the prompt, token for token. The moment one token differs, everything after it is a cache miss.
The consequence: order is design
That single rule turns prompt layout into an architectural decision. Put things in order of how rarely they change:
- The system prompt and instructions.
- Tool and function definitions.
- Few-shot examples and reference documents.
- Conversation history.
- The new user message and any freshly retrieved content.
csharp
var messages = new List<Message>
{
Message.System(StaticPolicy.Text), // identical bytes on every call
Message.System(ToolCatalog.Describe()), // sorted, deterministic
Message.User($"<context>{retrieved}</context>\n{question}") // changes every call
};
The stable material stays cacheable. The volatile material sits at the end, where changing it does not invalidate anything before it.
Things that silently break it
Almost every cache miss in production comes from one of a short list of habits.
- A timestamp or request id at the top of the system prompt. "Today is 27 October 2026, 09:44:12" makes the first token different on every call. Move it to the end, or drop the seconds.
- Non-deterministic serialization. Tool schemas serialized from a dictionary with unspecified key order produce different text on different runs. Sort them.
- Changing the tool set mid-conversation. Tool definitions are part of the prefix. Adding or removing one invalidates what follows it.
- Editing earlier turns. Summarizing or truncating the middle of a conversation rewrites the prefix. Appending is cheap, rewriting is not.
- Interleaving retrieved documents early. Placing per-query retrieval results before the stable examples pushes volatile content in front of stable content.
None of these is a bug in the usual sense. The output is correct. You just pay full price for it.
Measure it
The mechanics differ by provider, in minimum prefix length, entry lifetime and whether breakpoints are explicit or automatic, so read the documentation of the one you use. Then check what actually happens.
Every provider reports cache usage in the response. Anthropic returns the number of cache read and cache creation tokens, and OpenAI reports cached tokens inside its usage details. Log them.
csharp
var hitRate = usage.CachedInputTokens / (double)usage.InputTokens;
_metrics.RecordCacheHitRate(feature: "support-assistant", hitRate);
A hit rate per feature is one of the most actionable numbers in an LLM application. When it drops after a deploy, someone put something volatile near the top.
Conclusion
Caching rewards one discipline: keep the front of the prompt boring.
Treat the stable prefix as an interface that changes deliberately and rarely, version it like code, and keep everything that varies per request behind it. The effect is not a micro-optimization. For applications with large, repetitive prompts, it is often the largest single lever on both cost and latency.
References