AI glossary · Basic · Efficiency & infra
Precision (FP32, BF16, FP8)
Precision refers to how many bits an AI model uses to store and compute a single number. Formats such as FP32, BF16 and FP8 use fewer bits as you go down, which reduces memory and computation but makes the representation coarser.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra
- Added
- 2026-10-12
How it works
FP stands for floating point, and the number after it is the bit count. FP32 uses 4 bytes per number, FP16 and BF16 use 2 bytes, FP8 uses 1 byte, and FP4 uses 0.5 bytes. The bits are split into a part that handles 'the range of number sizes' and a part that handles 'fineness'. BF16 is the same size as FP16 but keeps a range as wide as FP32, so values overflow less often during training, and it is now used as something like the default for training and inference.
Bytes per number
Where you'll see it
You see it in the torch_dtype field of a Hugging Face model's config.json (e.g. bfloat16), in model cards, and in the dtype setting of inference engines. GPU spec sheets list different operations per second for each precision, and newer GPUs support lower precisions such as FP8 and FP4 in hardware. The 4bit and 8bit labels attached to the names of quantized models refer to the same thing.
Why it matters
Halving the precision halves the weight memory, and on GPUs that support it, computation also gets faster. This lets you fit a large model on a smaller GPU or handle more requests. In training, 'mixed precision', which does computation in low precision while keeping important values in high precision, is common. Recently, research on doing even training in FP8 and FP4 has been active.
Things to know
Lowering precision too far can reduce accuracy or make training unstable. Even at the same 8 bits, FP8 and INT8 (integer) work differently, so you need to check which one your engine and GPU support. Older GPUs do not support BF16 or FP8 in hardware, so you may not get the expected speed. Loading a model originally released in BF16 as FP32 just doubles the memory with almost no gain in quality.
Example
The 'TRACE' paper covered on this site deals with a method that builds quantization into the training process in advance to reduce the errors that arise when doing reinforcement learning on MoE language models in FP4. In practice, check whether torch_dtype in the model's config.json is bfloat16, and if your GPU supports FP8, consider switching to FP8 in the inference engine settings to save memory.