AI glossary · Advanced · Efficiency & infra · MLOps & LLMOps
Distributed training
Distributed training is a way of splitting the work of training a single model across multiple GPUs or multiple servers so they process it together. It is used when the model or training data does not fit on a single GPU, or when one GPU would take too long.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra, MLOps & LLMOps
- Added
- 2026-10-12
How it works
The most common approach is data parallelism. Every GPU holds a copy of the same model and trains on a different batch of data. Then the gradients each one computed (values that indicate which direction to adjust the model) are gathered and averaged, and all GPUs update the model identically. If the model is too large to fit on one GPU, model parallelism, which splits the model itself, is used as well. Sharding, which saves memory by having each GPU hold a portion of the weights, gradients and optimizer states (values the training tool keeps separately), is also widely used and often appears under the names FSDP or ZeRO.
Three ways to split the work
Data parallelismSplits the data
- Full model copy on each GPU
- Simplest and most common
- Model must fit on one GPU
Sharding (FSDP, ZeRO)Splits the training state
- Weights, gradients, optimizer states divided
- Saves a lot of memory
- More communication
Model parallelismSplits the model
- Layers or computation spread across GPUs
- Needed for very large models
- Most complex to set up
공통 All are ways for multiple GPUs to train one model together
Why it matters
Pre-training today's LLMs is practically impossible on a single GPU. During training, not only the weights but also the gradients, optimizer states and intermediate results must be held in memory, so training needs far more memory than inference. Distributed training is the basic technology for spreading this memory load and training time across multiple machines. When a model is large, even relatively small jobs such as fine-tuning require multiple GPUs.
Where you see it and how to get a feel for it
In practice, you encounter the term in the settings of training frameworks such as PyTorch DDP and FSDP, DeepSpeed and Megatron-LM. Options in training scripts such as the number of GPUs, world size and ZeRO stage are exactly the values that determine the distribution method. To get a feel for it, remember that "training memory is several times larger than weight memory." For common mixed-precision training, a frequently used rule of thumb is number of parameters × about 16 bytes ≈ memory for weights, gradients and optimizer states, with memory for intermediate results added on top.
Things to know
Doubling the number of GPUs does not double the speed. This is because communication time is spent exchanging results between GPUs, which is why the performance of interconnects such as NVLink and InfiniBand matters. When multiple servers are linked, the whole job can stop if just one of them stops, so checkpoints need to be saved frequently. Before preparing for distributed training, it is usually better in terms of cost to first check whether the job can be done on a single GPU with LoRA or quantization.
Example
Suppose a team wants to fully fine-tune a 7-billion-parameter (7B) model. By the rule of thumb above, 7 billion × 16 bytes ≈ 112GB is needed just for the weights, gradients and optimizer states, which does not fit on a single GPU with 80GB of memory (example). So the team weighs cost and quality to decide whether to rent one server with multiple GPUs and split these states using FSDP or DeepSpeed ZeRO, or to switch to LoRA and finish on a single GPU.