AI glossary · Advanced · Efficiency & infra
GPTQ·AWQ
GPTQ and AWQ are widely used quantization methods that shrink the weights of an already trained LLM to fewer bits, such as 4 bits. They make a model lighter without retraining, using only a small amount of sample data to limit quality loss.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra
- Added
- 2026-10-12
How it works
Both are post-training quantization methods applied after training is finished, and both make their decisions by passing a small amount of calibration data through the model. GPTQ converts the weights in each layer to low bits one at a time, and compensates for the resulting error by adjusting the other weights that have not yet been converted. AWQ (Activation-aware Weight Quantization) finds the important weights on the side where large input signals (activations) come in, and rescales them so they lose less precision. In the end, both store weights in 4 bits or similar and expand them again for computation.
The tricks behind GPTQ and AWQ
GPTQMaking up for errors
- Low bits, one at a time per layer
- Compensate errors with remaining weights
AWQProtecting important weights
- Weights with large activations treated as important
- Rescaled to lose less precision
공통 Both are post-training quantization that uses a small amount of calibration data without retraining
Where you'll see it
On Hugging Face you often see it in repositories whose model names end with tags such as '-GPTQ', '-AWQ' or '-Int4'. Inference engines such as vLLM have a quantization option in their settings, and a model's configuration file (config.json) lists quantization details such as the bit width and group size. A rough rule of thumb is 'number of parameters × 0.5 bytes (4 bits) ≈ weight memory'. Compared with the 16-bit original, weight memory drops to roughly a quarter.
Things to know
Quantization is not free. Quality loss can be more noticeable in delicate tasks such as math and coding, or in small models, so it is a good idea to compare against the original using real work data. If the calibration data is far from the actual use case, results can get worse. Also, the speed gain depends on which GPUs and inference engines support fast operations (kernels) for that format. GGUF is usually chosen for laptops and CPUs, while GPTQ and AWQ are often chosen for GPU servers.
Example
If a company wants to serve an open-weight model on a single GPU but the 16-bit original does not fit in memory, it can find the '-AWQ' version of the same model and launch it in vLLM with the quantization option turned on. It then compares answer quality against the original API's results on a set of frequently asked questions to judge whether it is usable.