Quantization and the KV cache: two ways to run models lighter

Why it was needed: memory runs out before compute

When running a large language model (LLM), the first wall you hit is memory rather than compute speed. The GPTQ paper paper points out that just storing the parameters of GPT3-175B in 16-bit floating point (FP16) takes 326GB. That is more than the memory (VRAM) of even the largest single GPU, so even inference required an expensive and complex setup tying together several GPUs.

Weights are not the only thing using memory. As a model generates text one token at a time, it stores the key and value vectors computed for each previous token so it does not have to recompute them. This store is the KV cache. The longer the conversation and the more requests handled at once, the larger the KV cache grows. The PagedAttention paper paper explains that the KV cache for each request is very large and grows and shrinks during generation, so poor management wastes a lot of memory through fragmentation (scattered empty space) and duplicate copies, which in turn reduces the number of requests that can be processed together (the batch size).

So making a model 'lighter' means answering two questions: how to store the weights compactly, and how to store the KV cache compactly and without waste. The first is answered mainly by quantization; the second by quantization and memory management techniques together.

Core idea: write with less precision, and pack without gaps

Quantization means reducing the number of digits used to write numbers. If weights written in 16 bits are written in 4 bits instead, the same model fits in roughly a quarter of the space. In exchange, the grid of representable values becomes coarser, so moving each original value to the nearest grid point introduces error. Most quantization research is about making this rounding error less harmful to model quality.

As an analogy, it is like cutting down on what you pack for a move. Quantization is swapping thick hardcovers for paperbacks to shrink the books themselves, while taking care that the text in a few important books does not get smudged. PagedAttention divides the bookshelf into fixed-size slots, puts each new book into any empty slot, and keeps only a list of where things are. Because you do not have to reserve a large block of space for each book in advance, less space is left empty.

The two approaches are not substitutes but complements. Quantize the weights to free memory for the KV cache, then quantize the KV cache itself and manage it in blocks, and the same GPU can handle longer contexts and more concurrent requests.

How it works: the weight side and the KV cache side

Weight quantization is usually applied in one pass to a model that has finished training. This is called post-training quantization (PTQ). GPTQ paper explains that retraining the model is too expensive at the scale of billions of parameters, so post-training methods that need only a little data and a few hours of compute are especially attractive for huge models. The flow is roughly as follows. ① Pass a small amount of calibration data through the model to see what each layer's inputs look like. ② Layer by layer, move the weights onto a low-bit grid. ③ Adjust the remaining values or change scales so that the error introduced shows up less in the layer output. ④ At inference, the GPU kernel reads in the low-bit weights and unpacks them again (dequantization) to compute.

The KV cache side runs in step with generation. In the prefill stage, when the prompt is first read, keys and values are created for all input tokens; then in the decoding stage, new keys and values are appended to the cache each time a token is generated. PagedAttention paper stores this cache in small blocks instead of one large contiguous chunk and tracks which blocks each request uses in a table. As a result, almost no empty space is wasted, and requests sharing the same prompt can even share blocks. On top of this, research on quantizing the KV cache values themselves to around 2 bits has recently been active.

Note that 'storing small' and 'computing fast' are not the same thing. The GPTQ paper states as a limitation that its speedup comes from reading compressed weights from memory faster, and that the multiplication itself does not get faster because mainstream hardware does not support mixed-precision multiplication such as FP16×INT4.

How it developed: GPTQ, AWQ, PagedAttention

GPTQ paper uses approximate second-order information (information on how sensitively errors propagate to the output) to compensate for the error from rounding one weight by adjusting other weights that have not yet been quantized. According to the paper, the only method previously applied at the 175B scale was simple round-to-nearest (RTN), which works fine at 8 bits but could not preserve accuracy below that. GPTQ compressed a 175B-parameter model to 3–4 bits in about 4 GPU hours while keeping accuracy loss small, and ran a 175B model on a single A100 for the first time. It reported end-to-end speedups over FP16 of about 3.25x on A100 and about 4.5x on A6000.

AWQ paper starts from the observation that 'not all weights are equally important.' Protecting the salient weights, 0.1–1% of the total, greatly reduces quantization error, and the key point is that finding these salient channels requires looking at the activation distribution (the input signal coming into that channel) rather than weight magnitude. In an experiment compressing OPT-6.7B to 3 bits, keeping just the salient 1% in FP16 improved perplexity (a language model metric where lower is better) from 43.2 to 13.0. However, mixing precisions is inefficient on hardware, so AWQ instead chose an equivalent transformation that scales up the salient channels in advance to reduce relative error. The AWQ paper points out that GPTQ can overfit to calibration data so that performance may wobble on data from other domains, and claims that its own method, which does not rely on backpropagation or reconstruction, generalizes better to instruction-following and multimodal models. Its accompanying inference framework, TinyChat, was reportedly more than 3x faster than the Hugging Face FP16 implementation.

PagedAttention paper targeted a different bottleneck of the same period: the KV cache. It created an attention algorithm inspired by virtual memory and paging in operating systems, and built the serving system vLLM on top of it. By cutting KV cache waste to nearly zero and enabling cache sharing within and across requests, it raised throughput 2–4x at the same latency compared with systems of the time such as FasterTransformer and Orca. The improvement grew with longer sequences, larger models and more complex decoding methods.

Where it is heading: where to spend a limited number of bits

Recent research is moving from 'the same number of bits everywhere' to mixed-precision allocation that 'gives more bits to the important places within a fixed budget.' The idea AWQ demonstrated for weights, 'protect the important few,' appears to be extending to the KV cache and MoE (mixture of experts) models.

PN-Mix paper starts from the problem that, when generating long outputs, KV cache memory and bandwidth become a severe bottleneck, and cutting to low bit widths such as 2 bits causes accuracy to drop sharply. Whereas earlier mixed-precision key cache methods chose important channels by looking only at key statistics, PN-Mix showed that, at the level of RoPE (rotary position embedding) channel pairs, errors in pairs with a large product of query and key magnitudes (norms) strongly disturb attention scores. These important pairs are selected once during prefill and reused throughout decoding. It reported accuracy comparable to or better than existing methods using 2.32–2.39 bits at an average of 2.26 bits, with performance degrading gradually even down to 1.5 bits.

MultiAlloc paper notes that the KV cache is the main consumer of memory in long contexts and complex reasoning, and argues that the real challenge in deployment is less compression itself than how to split a fixed memory budget between different strategies: 'raising per-layer precision' and 'preserving important tokens.' It uses lightweight offline profiling to measure layer sensitivity and token importance and fix an allocation policy in advance, which is then applied as-is during inference without further search. It reportedly outperformed existing methods consistently at the same memory budget on Mistral, Llama, DeepSeek-Qwen, QwQ and others. On the weight side, NatScale paper addresses MoE models. MoE uses only some experts per token but must store the weights of all experts, so allocation that gives more bits to sensitive experts matters. Arguing that expert rankings alone cannot determine how heavily to weight each error, NatScale proposed a framework that calibrates these weights, and across all nine settings on three models it achieved the lowest KL divergence (difference in output distribution) from the original model compared with five allocation baselines.

Limitations and open questions

First, saving storage does not by itself mean faster speed. As GPTQ paper noted, if the hardware does not support low-bit multiplication, the gain comes mainly from memory reads. AWQ paper built a separate inference framework because turning theoretical memory savings into measured speed requires additional engineering such as kernel fusion and packed weight layouts. Throughput (tokens/second) and TTFT (time to first token) should be measured directly on your own GPU and inference engine.

Second, the issues of calibration data and generalization remain. The AWQ paper pointed out that error-compensation methods can overfit to the calibration set. If a model will be used on a distribution different from the calibration data, such as internal company documents or a particular language, a quantized model should be evaluated separately on real work data rather than chosen on general benchmark scores alone.

Third, there is still no settled standard for measuring 'what is important.' PN-Mix paper showed what earlier methods that measured importance using only key statistics had missed, and NatScale paper argued that rankings alone cannot determine allocation. In other words, results differ even with the same bit budget depending on which importance signal is used. Conclusions can also differ by metric: NatScale had the lowest KL divergence in all nine settings but the lowest perplexity in only seven.

Fourth, risk grows nonlinearly as bit widths go down. According to the PN-Mix paper, the comparison baselines degraded sharply below 2.25 bits. The fewer the bits, the smaller the additional savings and the greater the risk of quality collapse, so in a service it is safer to first find the point where quality breaks down and set a comfortable margin above it.

Further study

If you are new to this, start with the introduction and related work sections of GPTQ paper. They lay out clearly why post-training quantization without retraining was needed for huge models, and what wall stood between 8 bits and 3–4 bits. Next, the introduction and Figure 2 of AWQ paper make the idea of 'finding the important 1% through activations' intuitive and let you compare where it took a different path from GPTQ.

For the KV cache side, first read the abstract of PagedAttention paper and the part of the paper analyzing where existing systems wasted memory. Once you understand why serving system throughput depends on memory management, it becomes natural to see why KV cache quantization research followed.

To follow recent work, read PN-Mix paper for channel-level mixed precision in the KV cache, MultiAlloc paper for the memory budget allocation problem, and NatScale paper for bit allocation in MoE weights, in that order. Reading the three papers through the single question of 'where to spend the same budget' makes each one's contribution clearer.