← Deep dives

Deep dives · 2026-11-18 · Advanced · By bombong

Mixed-precision quantization: different bits per layer, quality measured with KL divergence

Mixed-precision quantization does not reduce every layer of a model to the same number of bits. It gives more bits to sensitive parts and cuts more from insensitive ones. This article explains how to measure sensitivity as the difference between the next-token distributions of the original and quantized models (KL divergence). It also covers how the measured sensitivities are turned into an allocation problem within a memory budget, and the limits of that approach.

Contents
Key points
  • Reducing every layer to the same number of bits causes quality to drop sharply below an average of 4 bits. This creates an allocation problem: giving each layer a different number of bits.
  • KL divergence measures how far the probability distributions that the original model and the quantized model assign to the next token diverge. It captures changes across the whole output distribution more broadly than perplexity, which looks only at the single correct token.
  • Simply adding up sensitivities measured one layer at a time misses how layers affect one another (interactions) and how errors accumulate. Recent research is moving toward optimizing directly for model-level objectives.
  • The ranking of 'important layers' can change depending on how the calibration data is chosen and which metric is used. There are also reports that rankings based on perplexity have almost no correlation with reasoning performance.
  • The more finely bits are mixed, the more irregular the memory layout becomes, which can reduce hardware speed gains. So the allocation unit also needs to be designed to fit the hardware.

Why it was needed: the limits of reducing every layer to the same number of bits

Quantization is a technique that reduces memory by converting model parameters from high precision, such as 16 bits, to a lower bit count, such as 4, 3 or 2 bits. LLMs with billions of parameters are hard to fit as-is on on-device AI or hardware with little VRAM. That is why post-training quantization (PTQ), which compresses a trained model without retraining it, is widely used. GPTQ and AWQ are representative PTQ methods.

The simplest approach is uniform quantization, which reduces every layer to the same number of bits. But layers differ in how much the output breaks when their bits are reduced, that is, in their sensitivity. Some layers can be cut to 2 bits with almost no visible effect, while touching others even slightly makes the model produce nonsense. With the same memory budget, it is better to save bits on insensitive layers and concentrate them on sensitive ones. This is the starting point of mixed-precision quantization (MPQ).

The problem is that it gets harder as the average bit count goes down. CoopQ paper pointed out that existing mixed-precision methods struggle once average precision drops below 4 bits. The reason is that existing methods rely on metrics measured separately for each layer. Such metrics miss the interactions between layers that affect overall performance. So the field has two core questions. First, what should be used to measure quality damage? Second, how should bits be divided based on those measurements?

Core idea: concentrating a limited budget on the 'weak spots'

As an analogy, it is like reinforcing an old building with a fixed repair budget. Spending the same amount on every pillar wastes money on pillars that carry almost no load, while the pillars that actually hold up the building are under-reinforced. It makes more sense to first measure how important each pillar is and give more of the budget to the important ones. In mixed-precision quantization, the pillars are layers (or blocks, experts or tiles), and the budget is total memory, that is, the average number of bits.

This requires a ruler for measuring 'how much it broke.' The ruler covered in this article is KL divergence (Kullback-Leibler divergence). At every step, an LLM outputs a distribution assigning a probability to each possible next token. KL divergence puts the original model's distribution side by side with the quantized model's distribution and computes, as a single number, how much they diverge. It is 0 when the two distributions are the same, and it grows as the quantized model shifts probability toward tokens different from the original.

This ruler is useful because it does not need ground-truth data. Since the original model itself serves as the reference, a small set of unlabeled calibration sentences is enough. It also reflects not only the probability of the correct token but also how the probabilities of the second- and third-ranked candidates changed.

How it works: from measurement to allocation

The actual workflow generally has five steps (details vary by study). First, prepare calibration data. This is usually general text on the scale of a few hundred sentences. Second, run these sentences through the original model (e.g. BF16) and store the next-token probability distribution at each position. This is the baseline.

Third, measure sensitivity. The most common approach is to switch only one layer to a lower bit count, keep the rest at original precision, and average the KL divergence from the baseline distribution over the entire calibration data. Repeating this for each layer and each candidate bit count (e.g. 2, 3 and 4 bits) produces a table saying 'reducing this layer to this bit count causes this much damage.' Some methods use perplexity, gradients or quantization error instead of KL.

Fourth, use this table to solve for the optimal allocation within the budget. It becomes a combinatorial optimization problem: 'choose one bit count per layer so that total damage is minimized while total memory stays within the budget.' Tools such as integer linear programming (ILP), quadratic optimization and dynamic programming are used. Fifth, apply a quantization backend such as GPTQ with the chosen bits, then measure the KL and benchmarks of the finished model as a whole again.

This workflow has one hidden assumption: that adding up the damage measured separately for each layer gives the total damage, that is, additivity. In reality, errors in earlier layers propagate to and are amplified in later layers, so this assumption is not accurate. Much of the progression described below is about how to fix this assumption.

Why KL divergence: the trade-offs among quality metrics

Perplexity looks only at how high a probability the model gives to the correct token. It is easy to compute and intuitive, but as long as the correct token's probability stays about the same, large swings in the probabilities of other candidates do not show up well. KL divergence compares the entire distribution, so it catches these changes too. And because it uses the original model rather than a correct sentence as the reference, it aligns directly with the goal of quantization: 'how closely does it behave like the original?'

Research uses this signal as well. FlexQuant paper uses the model's perplexity entropy and KL divergence to set layer-wise mixed precision, and dynamically changes the bit count each time a token is generated. It reported a 1.3x end-to-end speedup while keeping accuracy loss negligible across various language tasks. The recent ACSeL paper uses RMQD (Relative Model Quality Degradation), a quality proxy based on the divergence between the next-token distributions of a candidate model and a reference model. The authors state that, across the model sizes and precision ranges they tested, RMQD is cheap to compute yet strongly correlated with downstream benchmark performance.

KL is not a cure-all, though. KL values are measured only on the contexts that appear in the calibration data. So they cannot guarantee how much abilities absent from that data, such as long mathematical reasoning, have been damaged. This is in the same vein as the problem the TASA study, covered in 'Where things are heading,' showed for perplexity-based sensitivity. In the end, with any metric, you also have to ask 'what data was it measured on?'

Three rulers for measuring quality

PerplexityCorrect-token probability

  • Easy to compute, intuitive
  • Requires correct sentences
  • Rankings may diverge from reasoning performance

KL divergenceDistribution compared with original

  • No labels; original is the reference
  • Reflects changes across the whole distribution
  • No guarantee for abilities outside calibration data

Model-level objectiveEvaluated on the whole assembled model

  • Reflects inter-layer interactions
  • Includes error propagation
  • High search cost; search space must be reduced

공통 All are used as the basis for deciding where to spend bits within a memory budget

A comparison of the main ways to measure sensitivity. All of them share the limitation of being measured on calibration data.

How the field developed: from looking at one layer at a time to the whole model

CoopQ paper framed the mixed-precision problem as a cooperative game among layers. It proposed SPQE, which efficiently estimates Shapley values from game theory (a method for fairly dividing each participant's contribution in a cooperation), to measure both per-layer sensitivity and inter-layer interactions. It turns these estimates into a binary quadratic optimization problem and assigns 2 or 4 bits to each layer under a strict memory constraint. On Llama-3, Gemma-2 and Qwen-3 with three PTQ backends (Quanto, HQQ, GPTQ), it reported 20–80% lower perplexity than the best baseline from an average of 4 bits down to 2 bits, with the gap widening as bits got tighter.

FlexQuant paper opened a dynamic direction in which the bit allocation is not fixed but changes during generation. SFMP paper targeted a different problem: existing mixed-precision methods either rely on expensive discrete optimization or lose hardware efficiency because of irregular memory layouts. SFMP extended bit counts to fractional values, turning discrete allocation into a continuous problem. On top of this it added block-wise mixed precision, row and column reordering to group important weights, and a unified GEMM kernel that supports any average bit count. It states that it outperforms layer-wise methods under the same memory constraint.

In MoE models, the allocation unit moved from layers to experts. GEMQ paper pointed out that existing methods estimate importance only within a layer and ignore how quantization changes the router's choices. So it handles the importance of experts across the whole model at once with linear programming, and lightly fine-tunes the router to fit the quantized experts. Around the same time, a joint pruning and quantization optimization study paper argued that most PTQ optimizes only per-layer error and misses how error accumulates and propagates through the network, and it proposed a mixed-precision strategy that directly reduces error propagation across the whole model. At 1–3 bits, it lowered WikiText perplexity by up to 21% compared with weight-activation quantization baselines. Compared with weight-only methods, it reported up to 59% lower on WikiText and up to 85% lower on C4.

In one line, the progression is this: at first, scores measured separately for each layer were added up, and later research moved toward accounting for inter-layer interactions and error propagation. At the same time, the allocation unit diversified from layers to blocks and experts.

Where things are heading

First, a trend of questioning the metrics themselves. TASA paper reported a phenomenon called the 'Perplexity Illusion.' The rank correlation between the layers flagged as important by perplexity-based sensitivity and the layers with the greatest effect on complex reasoning performance was Kendall τ≈0, meaning almost none. It also found that calibrating only on target-task data can actually hurt performance, and that mixing in general-domain data stabilizes sensitivity estimates. By jointly optimizing calibration data composition and bit allocation in this way, an average 3.5-bit model matched or beat several uniform 4-bit baselines. On GSM8K with LLaMA-3-8B, it scored more than 20 points higher than the strongest 3-bit baseline.

Second, a trend of directly optimizing model-level objectives. Q-Strata paper tackled the problem that, in MoE models, assigning bits to every linear layer of every expert makes the search space far too large, by splitting it into two stages. The inner stage ranks allocations within a block using a cheap proxy metric and stores the best candidates for each budget per block (the Pareto frontier) in advance. The outer stage only needs to choose one budget per block. Because the search shrinks that much, it can directly optimize an objective evaluated on the whole assembled quantized model, and it also captures the coupling between blocks that additive proxies miss. On three MoE models including Mixtral-8x7B-Instruct, it reported lower WikiText2 perplexity than uniform-bit GPTQ and GEMQ in the low-bit range.

Third, a trend of choosing the full set of deployment conditions together. ACSeL paper targets environments with tight memory and latency budgets, such as the edge. It treats model size, batch size, quantization algorithm and mixed-precision configuration as one joint selection problem. Among the possible combinations, it looks for the one predicted to give the highest quality while staying within budget. For this, it uses a lightweight surrogate model that predicts the distribution-divergence-based RMQD described earlier from the configuration alone, and performs layer-wise mixed-precision search with a structured search algorithm called BandPareto. This shows KL-style metrics expanding beyond bit allocation into a criterion for searching deployment configurations.

Limitations and open issues: what to consider in practice

When to use it. If uniform 4-bit gives sufficient quality, the benefit of mixed precision is small. As CoopQ paper results show, the gap widens as the average bit count gets tight below 4 bits. Conversely, it is well worth considering when memory forces you down to around 3 bits, or for MoE models where the many experts create a heavy memory burden.

The first common mistake is choosing calibration data that is skewed to one side. TASA paper results show that using only target-task data is also risky. The second is trusting a single metric. Whether perplexity or KL, it is an average measured on calibration data, so the final model must be rechecked on benchmarks for the tasks it will actually be used for. The third is mistaking the sum of per-layer scores for total damage. The studies that pointed out inter-layer interaction and error propagation paperpaperpaper make this point repeatedly.

Conflict with hardware is another major issue. As SFMP paper points out, mixing bits finely makes the memory layout irregular, so memory may shrink but speed may not improve as much as expected. Set the allocation unit to something hardware-friendly such as blocks, and first check that a kernel exists to run that bit combination. Search cost also has to be considered. The cost of measuring KL for every layer × bit combination grows with model size, which led to designs that eliminate search, like SFMP, or shrink the search space, like Q-Strata.

Finally, most reported numbers come from WikiText perplexity or specific benchmarks. There is no guarantee that the same rankings will hold on the actual distribution of queries in your own service.

Checks before adopting mixed precision

Need

  • Is memory insufficient with uniform 4-bit?
  • Is it a model with a heavy parameter burden, such as MoE?

Measurement

  • Does the calibration data include general-domain text?
  • Are you avoiding reliance on perplexity alone?
  • Are you avoiding mistaking the sum of per-layer scores for total damage?

Deployment

  • Is the allocation unit hardware-friendly?
  • Is there a kernel to run that bit combination?
  • Have you rechecked with benchmarks for the actual tasks?
The decision criteria from the article, organized as checklist items.

Further study

To get the concepts, start with CoopQ paper. It contains the field's basic framework in a single paper: the limits of per-layer metrics, interactions, and optimization under a budget. Reading FlexQuant paper, which uses KL divergence directly, next will give you a feel for 'measuring quality by distribution differences.' Then reading ACSeL paper shows how the same signal expands into deployment configuration search.

For a critical look at metrics, see TASA paper; for hardware efficiency and search cost, see SFMP paper. If you work with MoE models, learn expert-level allocation and router calibration from GEMQ paper, then continue with Q-Strata paper to see why two-stage allocation is needed. The perspective of error accumulating and propagating can be found in the joint pruning and quantization optimization study paper.

Related terms

Quantization GPTQ·AWQ Precision (FP32, BF16, FP8) MoE VRAM On-device AI Benchmark Local LLM

References

Foundational papers

Recent research

Written by AI from the listed references and reviewed. Please check the original sources for details.