AI glossary · Beginner · Efficiency & infra
VRAM
VRAM is the dedicated memory attached to a GPU; model weights and intermediate computation values must be loaded here for the GPU to compute quickly.
About this term
- Level
- Beginner · Core ideas for first-time readers
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra
- Added
- 2026-10-12
How it works
A GPU works quickly only with data in its own memory. So to run a model, all of its weights, the intermediate computation values and, for an LLM, the KV cache must fit in VRAM. If there is not enough room, a 'CUDA out of memory' error occurs, or part of the data is offloaded to CPU memory and things slow down sharply. High-performance AI GPUs use a fast memory called HBM as VRAM.
Getting a feel for it
Weight memory is estimated as 'number of parameters × bytes per number.' For example, loading a 7-billion-parameter (7B) model in 16-bit (2 bytes) takes about 14GB, and quantizing it to 4-bit cuts that to a quarter, about 3.5GB. On top of this, space is needed for the KV cache and computation headroom. The KV cache grows with longer context and with more requests processed at the same time.
How VRAM is shared
GPU memory (VRAM)
Model weightsKV cacheIntermediate valuesFree
Where you'll see it
You see it in the 'memory capacity' on GPU spec sheets, the GPU memory item in cloud instance descriptions, and the memory usage shown by nvidia-smi. The file sizes listed on a Hugging Face model page give a rough idea of how much VRAM the weights will take. Local LLM tools also often show the required memory when you choose a model.
Things to know
A GPU that can barely fit the weights is hard to use in practice. With long contexts or multiple users, the KV cache quickly takes up space. Training needs far more memory than inference, because gradients and optimizer states must also be loaded. Even if an MoE model has few active parameters, all of its weights must be loaded, so VRAM should be calculated based on total parameters.
Example
A team choosing an open-weight model for an internal chatbot first looks at the parameter count and weight file sizes on the Hugging Face model page. For example, running a 7B model in 16-bit takes about 14GB for the weights alone, so, allowing for concurrent users and long contexts, they choose a GPU with more VRAM than that or opt for a 4-bit quantized version.