AI glossary · Basic · Efficiency & infra · Language models
Prompt caching
Prompt caching is a technique that stores the computation results for the beginning of a prompt that is sent to an LLM identically every time, and reuses them in the next request. Because the same content is not recomputed from scratch, responses are faster and costs go down.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Infrastructure & using models
- Added
- 2026-10-08
How it works
When an LLM reads text, it creates intermediate computation results for each token and keeps them in memory in a form called the KV cache. Prompt caching keeps these results for a certain period even after the request ends. If the beginning of the next request is identical to the previous one, the stored results are loaded as is, and only the newly added question is computed. That is why the trick is to put content that does not change, such as the system prompt, long reference documents and tool descriptions, at the very beginning of the prompt.
Why it matters
Today's AI services often send long instructions and documents with every request, so recomputing the same content costs a lot of time and money. Prompt caching shortens the time until the first response appears. Several AI providers offer this feature by charging a lower rate for tokens read from the cache than for regular input. It is especially effective for chatbots with long conversations and for AI agents that keep using the same list of tools.
Things to know
The cache applies only to the 'identical part from the beginning' of the prompt. If content that changes every time, such as the date or the user's name, appears near the beginning, everything after it has to be recomputed. The cache usually expires after a certain time, so the benefit may be small for services with infrequent requests. Services differ in whether they apply caching automatically or require developers to specify it, and in how they charge for it, so check the documentation of the platform you use.
Example
Consider an internal chatbot that holds an entire company policy document and answers employees' questions. The policy document is the same every time and only the question changes, so if the document part is cached, from the second question onward only the new question needs to be computed. As more services rent large models in the cloud, as in the news covered on this site, 'AWS offers Z.ai's 753B MoE model GLM 5.3 on Amazon Bedrock,' whether such cost-saving features are supported becomes one of the criteria for choosing a model.
Where it came up on this site
- briefing AWS offers Z.ai's 753B MoE model 'GLM 5.3' on Amazon Bedrock 2026-10-07