AI glossary · Advanced · Language models
Attention
Attention is a method by which an AI model, when processing input such as a sentence, gathers information by giving more weight to other words that are closely related to the word it is currently handling. It is a core component of the transformer architecture.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Models & generation
- Fields
- Language models
- Added
- 2026-10-08
How it works
The input is first split into small pieces called tokens, and each token is converted into a bundle of numbers that carries its meaning (an embedding). Each token holds "the information I am looking for" (Query) and compares it with "this is the information I have" (Key) from other tokens to produce a score for how related they are. It then mixes in more of the content (Value) of high-scoring tokens to create a new Representation (the internal representation used inside the model). Doing this process from several perspectives at the same time is called multi-head attention.
Why it matters
Earlier approaches (RNNs) read words one at a time in order, so they easily missed relationships between words far apart. Attention can compare every position in a sentence with every other at once, so it understands long contexts better. Its computations parallelize well, so it runs fast on GPUs, and this is what made it possible to train huge LLMs. It is widely used not only in text models but also in image, video and audio models.
Things to keep in mind
Because it compares every pair of tokens, computation and memory grow quickly as the input gets longer. This problem makes it hard to extend the context window and led to optimization techniques such as the KV cache, which stores already-computed Keys and Values. Also note that a high attention score does not directly explain "why" the model made a decision. Alternatives such as attention variants with reduced computation and state space models are also being researched.
Example
For inference engines that run LLMs as a service, how efficiently they handle attention computation largely determines speed. News featured on this site, such as "vLLM v0.31.0 Released, Focused on Optimizing DeepSeek-V4.1-Flash Inference Performance," also ultimately relates to work on managing attention and the KV cache better in memory to make responses faster.