LLM infrastructure basics: GPU memory, serving and cost at a glance

What it is and why it matters: training and inference are different jobs

"Running" an LLM covers two different jobs. Training means adjusting the parameters (the numbers the model has learned) while the model sees large amounts of data. Inference means using the finished model to produce answers. Training also involves a forward computation followed by sending the error backward to update the values (backpropagation). So besides the weights, gradients, optimizer states and intermediate values must also be kept in memory. As a result, training a model takes far more GPUs than running inference on the same model. Megatron-LM paper also pointed out that memory limits make very large transformers hard to train. Its authors trained an 8.3-billion-parameter model on 512 GPUs.

Inference is not a one-time cost. It is an operating cost that comes with every user request. The vLLM paper paper cites an estimate that handling a single LLM request can cost 10 times more than a traditional keyword search. It argues that raising throughput to cut the cost per request has become important. Most companies do not train their own models. They serve existing models, which means running them so they can respond to users. For this reason, this article focuses on inference.

LLM inference has one key trait: it produces the next token one at a time, based on the tokens that came before (an autoregressive model). The vLLM paper explains that because tokens are generated in sequence, speed is limited by memory and the GPU's compute power is not fully used. FlashAttention paper also notes that on modern GPUs, compute speed has outpaced memory speed, so most transformer operations are slowed down by memory access. This is why much of the discussion about LLM infrastructure is about how to use memory efficiently.

Components: what GPU memory is used for

GPU memory (VRAM) holds three main things. The first is the weights. They are the model itself and stay fixed for the whole time it is being served. The second is the KV cache. It is a memo that stores the attention keys and values of tokens already processed, so the same computation does not have to be repeated when generating the next token. A separate cache is created for each request; it grows as the text gets longer and disappears when the request ends. The third is activations. These are intermediate values that exist briefly during computation.

The vLLM paper paper gives an example of real proportions. When serving a 13B (13 billion parameter) model on a 40GB A100 GPU, the weights take about 65% of memory, the KV cache takes close to 30%, and activations and other items use the remaining roughly 5%. The weights do not change and the activations are small. So how the KV cache is managed determines how many requests can be processed together at once (the batch size). According to the vLLM blog source, on LLaMA-13B the KV cache for a single sequence alone can reach up to 1.7GB.

The size of the weights is easy to estimate. The Hugging Face guide source gives this rule of thumb: for a model with X billion parameters, you need about 4X GB in FP32 and about 2X GB in BF16 or FP16. For example, Llama-2-70b needs 140GB and Falcon-40b needs 80GB. The guide explains that for short inputs under 1,024 tokens, the memory required is determined almost entirely by the weights. In practice, bigcode/octocoder, estimated at 31GB in BF16, measured about 29GB when loaded. Keep in mind that with longer inputs or more concurrent requests, the KV cache's share grows accordingly.

Precision and quantization: how finely to write down numbers

Parameters are decimal numbers, and precision is how many bytes are used to write down each one. FP32 uses 4 bytes, FP16 and BF16 use 2 bytes, and FP8 uses 1 byte. BF16 is the same size as FP16 but can represent a wider range of numbers, so values are less unstable during training. The Hugging Face guide source explains that nearly all models today are trained in BF16. It also notes that running the earlier octocoder in FP32 would have required 64GB.

Quantization stores weights with fewer bits, such as 8 or 4 bits. The Hugging Face guide source cites research showing that lowering precision to 8 or 4 bits gives computational benefits without a large drop in performance. Counting bytes, weights take about X GB per X billion parameters at 8 bits and about X/2 GB at 4 bits. For a 70B model, this gives an estimate of about 35GB for the weights alone at 4 bits. According to the vLLM documentation source, the engine supports quantization formats such as FP8, INT8, INT4, GPTQ/AWQ and GGUF. GPTQ and AWQ are weight quantization methods, and GGUF is a file format commonly used for local execution.

There are two things to watch out for. First, the effect of quantization on quality varies by model and task. So you must check it with an evaluation set from your own work. Second, shrinking the weights does not shrink the KV cache. For services that handle long documents or many concurrent requests, memory can still run short even after the weights are reduced.

Inference engines and continuous batching: balancing latency and throughput

An inference engine is server software that keeps a model loaded in memory and efficiently handles requests from many users. Each request goes through two phases. First, in the prefill phase, the whole prompt is processed at once, and the first token is produced. The time taken up to this point is TTFT (time to first token). In the following decode phase, tokens are generated one at a time. Users feel the TTFT and how fast text appears after that. Operators look at the number of tokens the GPU produces per second, that is, throughput (tokens/s). Throughput is directly tied to cost.

Latency and throughput pull against each other. Batching many requests together lets the weights be read from memory once to serve multiple requests, so throughput rises. But as the batch grows, each request may wait longer or each step may slow down, which can increase latency. In older approaches, a new request could only join after the whole batch had finished. Continuous batching removes finished requests and immediately inserts waiting requests at every decode step, reducing the time the GPU sits idle. The vLLM documentation source lists continuous batching along with chunked prefill (splitting the prefill of long prompts into pieces), prefix caching (reusing identical leading parts) and disaggregated prefill/decode execution as supported features.

The core technology of vLLM is PagedAttention. According to the vLLM paper paper, existing systems reserved contiguous memory in advance for each request equal to the maximum length (e.g. 2,048 tokens). As a result, only 20.4–38.2% of KV cache memory actually held token information. PagedAttention, like an operating system's virtual memory, splits the KV cache into fixed-size blocks and allocates blocks as needed. The vLLM blog source states that this cuts waste to under 4%, and saves up to 55% of memory when generating multiple answers from one prompt. The paper reported 2–4x higher throughput than FasterTransformer and Orca at the same level of latency, and the blog reported up to 24x higher throughput than HF Transformers and up to 3.5x higher than TGI. However, these multiples are experimental results from specific models, GPUs and data.

There are also techniques that speed up the attention computation itself. FlashAttention paper splits attention into small tiles and computes them in the fast SRAM on the GPU chip. This reduces the number of reads and writes to the relatively slow HBM (the GPU's high-bandwidth memory). While computing exact attention rather than an approximation, it made attention computation up to 7.6x faster on GPT-2 and reduced memory use to a level proportional to sequence length. The vLLM documentation source states that it uses optimized attention kernels, including FlashAttention.

When a model doesn't fit on one GPU: tensor parallelism and pipeline parallelism

The Hugging Face guide source noted that the A100 and H100, the largest GPUs at the time of writing, have 80GB of memory. It explains that most of the models it uses as examples exceed 80GB just to load, so tensor parallelism or pipeline parallelism is needed. In practice, it is simpler to first check whether quantization can fit the model on one GPU before adding more GPUs.

Tensor parallelism splits the large matrices inside each layer across multiple GPUs, computes the pieces separately, and then combines the results. Megatron-LM paper implemented this kind of intra-layer model parallelism in PyTorch by adding only a few communication operations. This approach can be combined with pipeline parallelism. The study sustained 15.1 PetaFLOPs on 512 GPUs, achieving 76% scaling efficiency compared with a single GPU. Because tensor parallelism requires GPUs to exchange results at every layer, the connection between GPUs (the interconnect) must be fast. For this reason, GPUs within the same server are usually grouped together.

Pipeline parallelism cuts the layers in order, placing the earlier layers on GPU A and the later layers on GPU B. It requires little communication. However, the Hugging Face guide source points out that naive pipeline parallelism does not solve the problem of the later GPU sitting idle while the earlier GPU works, so more advanced methods are needed. According to the vLLM documentation source, the engine supports tensor, pipeline, data, expert and context parallelism for distributed inference. Of these, data parallelism runs multiple replicas of the model and is used to increase the throughput of a model that fits on one GPU.

Step by step: choosing a deployment option and estimating cost

If you are just starting, the following order is recommended. First, write down your requirements. Decide the number of requests per day, the average input and output tokens per request, the acceptable TTFT, and whether data may be sent outside. Second, estimate memory. Calculate the weights as 'parameters × bytes' and leave headroom for the KV cache. Third, choose a deployment option. Fourth, calculate the cost. Fifth, measure TTFT and throughput with a load test and adjust the batch size and number of GPUs.

There are four main deployment options. A cloud API means paying a model company for the tokens you use through its API. It has the lowest operational burden, but data leaves your organization. A GPU cloud means renting GPUs by the hour and running an inference engine yourself. You can freely choose open-weight models, but you have to handle operations yourself. Self-hosted servers require buying hardware, power and operations staff. In return, data stays internal, and this option pays off when usage is consistently high. Local execution (local LLM) means running a small or quantized model on a PC or laptop, which suits personal testing. vLLM can run a server compatible with the OpenAI API sourcesource. So when moving from an API to self-operation, you need to change less application code.

Estimate cost as follows. For an API, 'monthly cost ≈ monthly requests × (average input tokens × input price + average output tokens × output price)'. Input and output are often priced separately, so calculate them separately. For example, with 1,000 requests a day, each with 2,000 input tokens and 500 output tokens, that is 2 million input tokens and 500,000 output tokens per day. Over 30 days this comes to 60 million input tokens and 15 million output tokens, which you then multiply by each price.

If you operate it yourself, think of it as 'cost per token ≈ GPU cost per hour ÷ tokens processed per hour'. If throughput doubles, the cost per token halves. According to the vLLM blog source, LMSYS adopted vLLM and cut the number of GPUs needed to handle the same traffic by 50%. On the other hand, even if the GPU sits idle because there are few requests, the hourly cost is still charged. So when usage is low or fluctuates, an API tends to be better, and when usage is consistently high, self-operation tends to be better.

Checklist and common mistakes

Before deployment, check three groups of items. For memory, check whether you calculated the weights using the byte count for their precision, whether you left headroom for the KV cache, and whether you evaluated quality after quantization. For performance, check whether you set a TTFT target, whether you measured throughput (tokens/s) under real load, and whether you are using an inference engine that supports continuous batching. For cost and operations, check whether you calculated input and output tokens separately, whether you are monitoring GPU utilization, and whether you confirmed internal company rules for sending data outside.

Common mistakes include the following. Counting only the weights and forgetting the KV cache, so memory runs out on long requests. Expecting benchmark multiples such as 'up to 24x' to hold unchanged in your own environment. Pushing only for throughput until TTFT increases and the user experience gets worse. Using multiple GPUs with naive pipeline parallelism and leaving them idle most of the time. Bringing in self-hosted servers despite low request volume, so GPU utilization stays low.

You should also know the limitations. The figures in this article come from sources at a specific point in time. For example, the statement that 'the largest GPU has 80GB' reflects the time the Hugging Face guide source was written. Hardware and engines change quickly, so when making actual decisions, recheck against the latest documentation and your own measurements.

Further study

To start, reading the Hugging Face guide source is recommended. It walks through how to calculate weight memory, precision, FlashAttention, and architectural improvements such as MQA and GQA, step by step with code examples. Next, it helps to learn from the diagrams in the vLLM blog source how PagedAttention manages the KV cache in blocks.

To go deeper into the principles, read the vLLM paper paper. It covers in detail where memory waste comes from and how the scheduling is designed. FlashAttention paper is good for understanding the memory hierarchy inside a GPU, HBM and SRAM. If you want to know the starting point of model parallelism, look at how Megatron-LM paper implemented tensor parallelism.

To try it yourself, check the continuous batching, quantization and parallelism options in the feature list of the vLLM documentation source, and use the Quickstart guide to run a small model first. Once it is running, measuring TTFT and throughput yourself is the best way to learn.