AI Trend Research

← AI glossary

AI glossary · Advanced · Efficiency & infra

PagedAttention

PagedAttention is a memory management technique that splits the KV cache, which builds up while an LLM generates an answer, into small blocks (pages) and places only as many as needed into free spots in GPU memory. The inference engine vLLM made this technique widely known.

About this term
Level
Advanced · Specialist terms that need the inner workings
Added
2026-10-12
In plain words Instead of reserving a whole row of seats in advance for a group, it is like assigning empty seats anywhere for however many people actually show up and tracking their locations with a seating chart. The idea comes from the way computer operating systems manage memory in page-sized units.

How it works

The KV cache stores the computed results for the tokens read and written so far, so it grows as the answer gets longer. The older approach reserved a contiguous block of memory for the maximum length of each request in advance, so when an answer ended early, that space was simply wasted. PagedAttention splits the KV cache into blocks of a fixed number of tokens and hands out one free block at a time as needed. Even if the blocks are scattered around memory, it records their locations in a block table and looks them up during the attention computation.

Managing scattered blocks with a block tableRequest A → Block table; Request B → Block table; Block table → Block · A; Block table → Block · B; Block table → Block · A; Block table → Free block(When needed)GPU memory · KV cache spaceRequest AAnswer growing longerRequest BShort answerBlock tableBlock locations perrequestBlock · ABlock · BBlock · AFree blockFor the next requestWhen needed
Managing scattered blocks with a block table Each request gets only as many blocks as it needs, and the block table remembers where each block is. It is fine for blocks to be scattered across memory.

Why it matters

With less wasted memory, more requests can be loaded onto the same GPU at once. Combined with continuous batching, this raises throughput significantly. It also makes it easier for requests that start with the same system prompt to share their leading blocks, which is the basis for server-side prompt caching (prefix caching). Today, many inference engines manage the KV cache in a similar way.

Things to know

You will rarely implement it yourself; you encounter it through the number of KV cache blocks printed in the logs when starting vLLM, the GPU memory utilization ratio (gpu-memory-utilization), and the block size setting. For a rough sense, think of it as "GPU memory − model weights ≈ space available for the KV cache." The larger this space, the more concurrent requests and the longer the context length you can handle. PagedAttention only reduces wasted memory; it does not make the KV cache itself smaller, so with very long contexts you can still run out of memory.

Example

Suppose a developer starts a model with vLLM on a single GPU and gets an error saying there is not enough KV cache space for the maximum context length listed on the model card. Looking at the number of KV cache blocks in the logs, they reduce the maximum context length to what the service actually needs, or raise the GPU memory utilization ratio. If that is still not enough, they switch to a quantized version of the model to shrink the space taken by the weights and free up that much more room for KV cache blocks.

Related terms

KV cache Continuous batching Inference engine Prompt caching VRAM Attention