KV cache

How it works

When a Transformer model generates a new token, it refers to all previous tokens through a computation called attention, and in doing so it computes values called Key and Value for each token. First, it reads the entire prompt at once to produce these values and stores them in the KV cache. Then, as it generates tokens one by one, it pulls the values from the cache and computes only the Key and Value of the newly generated token, appending them to the cache. As a result, it does not have to recompute the earlier part every time.

Why it matters

Without a KV cache, the model repeats the same computations as the answer grows longer, which slows things down sharply. The cache is what lets a chatbot keep producing long answers quickly. On the other hand, the cache takes up a lot of GPU memory, so the longer the context window or the more concurrent users, the more memory becomes the bottleneck. That is why many techniques for cutting inference cost and latency focus on how to use the KV cache more sparingly.

Things to know

A KV cache is not something the model learns; it is temporary storage for reusing computation within a single conversation or request. There are various techniques for saving memory, such as quantization that shrinks the cache into smaller number formats, managing memory in fixed-size blocks, and dropping older parts. When multiple requests share the same beginning, that cache can be reused, which is the basis of prompt caching. Shrinking the cache too much can degrade answer quality, so a balance is needed.

Example

The throughput of inference engines that serve LLMs varies greatly depending on how efficiently they manage the KV cache. As with the news covered on this site, 'vLLM v0.31.0 released… focused on optimizing inference performance for DeepSeek-V4.1-Flash', updates to open-source inference engines often focus on running new models faster and more cheaply, and KV cache management is one of the key factors. In video generation research, how to manage the KV cache when generating long videos is also a research topic.

Where it came up on this site

• Papers TRACE: Rollout-based QAT (quantization-aware training) for FP4 reinforcement learning of MoE language models 2026-10-08
• briefing vLLM v0.31.0 released… focused on optimizing inference performance for DeepSeek-V4.1-Flash 2026-10-07
• Papers In-Distribution Forcing: in-distribution KV management for generating long videos at test time 2026-10-07

Related terms

Attention Transformer Inference Prompt caching Context window Speculative decoding