AI glossary · Advanced · Efficiency & infra
Model parallelism
Model parallelism is a way to split a large model that does not fit on a single GPU across multiple GPUs and have them compute together. The main approaches are tensor parallelism (TP), which splits the computation within a layer, and pipeline parallelism (PP), which splits the layers in sequence.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra
- Added
- 2026-10-12
How it works
In tensor parallelism, several GPUs each take a piece of the large matrix computation within a layer and combine the results. Because GPUs must exchange results at every layer, communication is very frequent. In pipeline parallelism, the earlier layers sit on one GPU and the later layers on another, and data is passed along in order. To keep the later GPU from sitting idle while the earlier one works, the input is split into small batches (micro-batches) and fed through. MoE models also use expert parallelism, which distributes the experts across GPUs.
Tensor parallelism vs pipeline parallelism
Tensor parallelism (TP)Splits computation within a layer
- Every GPU takes part in every layer
- Very frequent communication
- Suited to one server with fast links
Pipeline parallelism (PP)Splits layers in sequence
- Each GPU computes only its group of layers
- Communication only at stage boundaries
- Idle GPU bubbles occur
공통 Both split a model that does not fit on one GPU across multiple GPUs
Where you'll see it
The most common place is inference engine settings. In engines such as vLLM, the tensor-parallel-size and pipeline-parallel-size options determine how many GPUs to split across and how. On the training side, they appear as the abbreviations TP and PP in the settings of frameworks such as Megatron-LM and DeepSpeed. Model cards or serving guides for large open-weight models sometimes list recommended setups such as 'N GPUs, TP N'.
Getting a feel for it
First, estimate 'number of parameters × bytes per number ≈ weight memory'. For example, loading a 70-billion (70B) parameter model in BF16 (2 bytes per number) takes about 140GB for the weights alone, which does not fit on a single GPU. Add room for the KV cache on top of that to decide how many GPUs you need. A common rule of thumb is to use tensor parallelism within a single server whose GPUs are connected by fast NVLink, and pipeline parallelism when spanning multiple servers.
Good to know
The more you split, the more GPU-to-GPU communication cost you incur, so if the model fits on one GPU, not splitting is usually more efficient. Running tensor parallelism across servers connected by a slow network can actually make things slower due to waiting on communication. Pipeline parallelism tends to create idle GPU time between stages (bubbles). Shrinking the model with quantization to reduce the number of GPUs needed is often the cheaper solution.
Example
Suppose a team wants to serve a 70B-class open-weight model in-house. Since the weights alone are about 140GB in BF16, they choose a single server with four 80GB GPUs connected by NVLink (an example) and launch vLLM with tensor-parallel-size 4. If only two GPUs are available, they also consider switching to a 4-bit quantized version to reduce memory.