AI glossary · Advanced · Efficiency & infra
FlashAttention
FlashAttention is a computation method that restructures the Transformer's attention calculation to fit the GPU's memory layout, making it faster and using less memory while leaving the results unchanged.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra
- Added
- 2026-10-12
How it works
Attention calculates how related every pair of tokens in a sentence is, and a naive implementation repeatedly writes and reads a huge intermediate table to and from GPU memory (HBM). On GPUs, this back-and-forth with memory is often the bottleneck rather than the computation itself. FlashAttention splits the input into small tiles, finishes the computation in the small, fast memory on the GPU chip (SRAM), and never stores the huge intermediate table in full. The key point is that it is exact attention that produces the same result, not an approximation.
Naive attention vs FlashAttention
Naive attentionStores the whole intermediate table
- Writes a large intermediate table to GPU memory
- Frequent memory round trips
- Long inputs quickly exhaust memory
FlashAttentionSplits into tiles and computes on-chip
- Uses fast on-chip memory
- Fewer memory round trips
- Can handle longer inputs
공통 Both are exact computations that produce the same attention result
Why it matters
Lower memory use allows training and inference with longer contexts, and speed improves as well. This is why it is counted among the technologies supporting today's LLMs with long context windows. PyTorch, Hugging Face Transformers and major inference engines use this kind of optimized attention computation by default or offer it as an option. Successive versions have also brought improvements tailored to new GPU generations.
Things to keep in mind
You will run into this name in the attn_implementation option when loading a model with Hugging Face Transformers, or in the flash-attn package during installation. Support depends on the GPU generation and precision (FP16, BF16, etc.), so on older GPUs it may fail to install or be replaced by a different attention implementation. It is not a technique that improves model quality; it performs the same computation faster and with less memory. The effect is especially large during training or when reading long prompts, and for mostly short conversations the noticeable difference may be small.
Example
Suppose a team fine-tuning a model to summarize long reports on in-house GPUs hits an out-of-memory error as soon as it increases the input length. The team first checks whether its GPU generation supports FlashAttention, installs the flash-attn package, and then changes the attention implementation option when loading the model. With results unchanged and memory use reduced, it can continue training with longer inputs on the same GPUs.