AI Trend Research

← AI glossary

AI glossary · Basic · Efficiency & infra

HBM

HBM (High Bandwidth Memory) is memory built by stacking several memory chips on top of each other like an apartment building and placing them right next to a compute chip such as a GPU, so that very large amounts of data can be exchanged at once.

About this term
Level
Basic · Technical terms you often meet in AI news
Added
2026-10-08
In plain words If ordinary memory is like a distant warehouse connected by a narrow two-lane road, HBM is like building a high-rise warehouse right next to the factory and opening a wide passage with hundreds of lanes. However, this high-rise warehouse is difficult and expensive to build, so it cannot be built just anywhere.

How it works

HBM stacks DRAM chips in several layers and drills very small holes through the chips to connect the layers directly with electrical pathways (this is called TSV, through-silicon via). This stack is placed on the same base board as a compute chip such as a GPU, right beside it. Because the distance is short and there are a great many connection paths, the amount of data that can flow at once, that is, the bandwidth, is much larger than ordinary memory. The numbers attached, as in HBM3, HBM3E and HBM4, indicate the generation; higher generations are generally faster and hold more.

HBM's stacked structureDRAM layer → DRAM layer(TSV); DRAM layer → DRAM layer(TSV); DRAM layer → Base die(TSV); Base die → GPU compute chip(Wide passage); Distant ordinary memory → GPU compute chip(Slow)HBM stackDRAM layerTopDRAM layerDRAM layerBase dieBottom chip of thestackGPU compute chipPlaced right beside itDistant ordinarymemoryNarrow passageTSVTSVTSVWide passageSlow
HBM's stacked structure The column on the left is an HBM stack with several layers of DRAM chips, and the layers are connected directly by TSVs. The stack sits right next to the GPU and exchanges data through a wide passage (the number of layers is illustrative).

Why it matters

Large AI models must constantly fetch huge numbers of parameters and intermediate results from memory. No matter how fast the compute chip is, if data arrives slowly from memory, the chip sits idle waiting; this is commonly called the 'memory bottleneck.' Because HBM reduces this bottleneck, it has become a practically essential component of GPUs and accelerators for AI training and inference. As a result, along with the AI boom, the earnings and competition of memory companies that make HBM have become a major focus in the industry.

Things to know

HBM requires demanding processes to stack and connect chips precisely, so it is much more expensive than ordinary memory and its production is hard to scale up quickly. For this reason, HBM supply is sometimes cited as a cause when AI chips are in short supply. Also, since it must sit right next to the compute chip, capacity cannot be increased freely, and heat management is an important challenge. Smartphones and ordinary PCs mainly use other types of memory instead of HBM because of cost and power.

Example

Most GPUs and AI accelerators used in AI data centers come with several HBM stacks attached beside the chip. Among the news on this site is a report that AMD CEO Lisa Su said the company has begun shipping products equipped with HBM4. Whenever a new generation of AI chips comes out, which generation of HBM it uses and how much is presented as a key specification. This is also why earnings news from memory companies that make HBM, such as Samsung Electronics, is covered as AI industry news.

Where it came up on this site

Related terms

GPU NPU Parameter KV cache Inference LLM