AI glossary · Basic · Language models
Transformer
The Transformer is a neural network architecture built around attention, a mechanism that weighs how related the words (tokens) in a sentence are to each other, all at once. Most LLMs today are based on this architecture.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Models & generation
- Fields
- Language models
- Added
- 2026-10-08
How it works
First, the text is split into small pieces called tokens, and each token is turned into a bundle of numbers (an embedding). Position information, which tells the model where each token sits, is added to this. Next, in the attention step, each token looks at the other tokens in the sentence and gathers relevant information, and the following feed-forward step processes that information further. Stacking and repeating these two-step layers (blocks) dozens of times gradually deepens the understanding of context, and at the end a generative model predicts the next token.
Why it matters
RNNs, widely used before the Transformer, processed text one word at a time from the beginning, which made long text hard to handle and processing slow. The Transformer can process a whole sentence at once, which made it well suited to splitting large computations across GPUs. This made it possible for performance to keep improving as models and data were scaled up, and it became the foundation of today's LLMs. The Transformer architecture is widely used not only for text but also in areas such as images, speech and robot control.
Things to know
Because attention compares every pair of tokens, compute and memory grow quickly as the input gets longer. Various improvements are emerging to reduce this burden, such as the KV cache, MoE and hybrids with state space models. Also, 'Transformer' refers to an architecture, so two Transformers can differ greatly in capability depending on what data they were trained on and how.
Example
Most LLM services, such as chatbots, translators and coding assistants, are built on the Transformer. Researchers keep modifying this architecture. 'ALoDLM', covered on this site, is a looped diffusion language model that adjusts loop depth per token; instead of processing every token the same number of times, it tries to repeat the computation only as many times as each token needs.