AI glossary · Advanced · Efficiency & infra
Quantization
Quantization is a technique that makes an AI model smaller and faster by representing the numbers inside it (parameters, etc.) with fewer bits. It slightly lowers precision in exchange for large reductions in memory and compute cost.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra
- Added
- 2026-10-08
How it works
A model's parameters are often stored in 16-bit or 32-bit formats that hold decimal numbers precisely. Quantization converts these numbers into shorter formats such as 8-bit or 4-bit, reducing the number of slots for values that can be represented. Each original value is rounded to the nearest slot, and a reference value that sets the spacing between slots (the scale) is stored alongside so the original magnitude can be roughly restored later. There are two approaches: converting a model directly after training is finished (PTQ), and building quantization into training in advance so the model gets used to the error (QAT, quantization-aware training).
Why it matters
LLMs have so many parameters that running them as-is requires expensive GPUs and a lot of memory. Quantization lets the same model fit in less memory, so one machine can run a larger model or serve more users at once. It is also close to essential for on-device AI, where AI runs on a laptop or smartphone. When an open-weight model is released, quantized versions are often distributed alongside it.
Things to keep in mind
Cutting too many bits can lower answer quality or cause more mistakes on certain tasks. When some values inside a model are unusually large, rounding errors grow, so various techniques are used to handle those values separately. Even at the same bit width, quality differs by method, so it is a good idea to evaluate a quantized model separately from the original. Also remember that you only get a real speed gain if the hardware supports the number format in question.
Example
People who run open-weight LLMs on their own PCs often download versions quantized to 4 or 8 bits. On the research side, there are also attempts to extend quantization to the training stage. A paper featured on this site, 'TRACE: Rollout-based QAT (quantization-aware training) for FP4 reinforcement learning of MoE language models', covers a method that applies QAT so that MoE models train well during reinforcement learning even while using a 4-bit number format (FP4).