Attention and the Transformer: reading a whole sentence at once

Why it was needed: the bottleneck of reading one word at a time

Before the Transformer, the most widely used architecture for translation and language modeling was the RNN (recurrent neural network). An RNN reads a sentence one token at a time from the start, passing a 'hidden state' that summarizes what it has read so far to the next step. The Transformer paper paper pointed out that this approach is inherently sequential, so computation within a sentence cannot be split and run in parallel, and this becomes critical as sentences get longer. Memory limits also make it hard to batch many sentences together.

Some translation models used CNNs (convolutional neural networks) to reduce sequential computation. CNNs can compute all positions in parallel, but according to the same paper, the number of operations needed to connect two distant words grows with the distance (linearly for ConvS2S and logarithmically for ByteNet). The longer the connecting path, the harder it is to learn relationships between distant words.

Attention itself was already in use. It linked the encoder and decoder as a mechanism that let a translation model choose which part of the source to draw on. According to the paper, however, it was used together with RNNs in almost all cases. The Transformer went a step further and asked: 'What if we remove the RNN and keep only attention?'

Core idea: deciding whom to listen to in a meeting room

Attention can be compared to a meeting. Each word in the sentence sits in the meeting room as one person. Before refining what they will say, each person looks around at the others, taking in more from people relevant to their own point and less from those who are not. The word 'it', for example, pays attention to 'the animal' earlier in the sentence. Once everyone has mixed the others' words with weights in this way, each word has a new representation that reflects its context.

This process is usually explained with three parts: Query (question), Key (label) and Value (content). Each word has 'what I am looking for' (Query), 'a label describing me' (Key) and 'the content I actually hand over' (Value). The model scores how well my Query matches each other word's Key, turns the scores into proportions that sum to 1, and mixes the Values according to those proportions. Words within one sentence referring to each other this way is called self-attention. The paper defines it as 'an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence.'

The advantages are clear. Because every word looks at every other word at once, the number of operations to connect any two words is constant no matter how far apart they are, and all positions can be computed at the same time. As the paper itself acknowledged, however, taking a weighted average over many words can blur detail. To address this, the paper introduced multi-head attention, which runs several attention operations side by side.

How it works: stacking encoder and decoder layers

The original Transformer was designed for translation and is split into an encoder that reads the input sentence and a decoder that writes the output sentence. Processing roughly follows these steps. First, the input sentence is split into tokens, and each token is turned into a numeric vector, called an embedding. Because attention does not know word order on its own, position information is added here. The paper's original model used position representations that are not learned (have no parameters).

Second, the encoder is a stack of six identical layers. Each layer has two sub-layers. One is multi-head self-attention, and the other is a simple fully connected neural network applied to each token separately (a feed-forward network). Each sub-layer has a residual connection that adds its input directly to its output, followed by layer normalization. To make this work, the output size of every sub-layer was set to 512 dimensions.

Third, the decoder also stacks six layers but has one extra sub-layer. The decoder's self-attention is masked so that each position cannot see positions after it, because it must not peek at future words that have not been written yet. The extra sub-layer is attention over the encoder's output, which decides which part of the source to draw on during translation (commonly called cross-attention).

Fourth, the decoder produces output one token at a time. It works as an autoregressive model, feeding the token it just produced back into the input to predict the next token. During training, the entire target sentence can be fed in at once together with the mask, allowing parallel processing. This is why the Transformer trains quickly.

Starting point: what the 2017 Transformer paper introduced

'Attention Is All You Need' paper is a paper published in 2017 by researchers at Google Brain and Google Research. By its own account, its key point is that it is the first transduction model to compute representations of its input and output using only self-attention, without recurrence or convolution. According to the contributor notes, scaled dot-product attention, multi-head attention and parameter-free position representations were proposed in this paper.

The results were strong. On WMT 2014 English→German translation, it scored 28.4 BLEU (a translation quality score), beating the previous best results, including ensembles combining several models, by more than 2 BLEU. On English→French translation, it set a single-model record of 41.8 BLEU after training for 3.5 days on eight GPUs. The paper stated that this training cost was 'a small fraction' of that of the previous best models. It also noted that training for just 12 hours on eight P100 GPUs reached state-of-the-art translation quality at the time.

The paper also extended its tests beyond translation. Applied to English constituency parsing, it worked well with both large and small amounts of training data. The source material for this note does not include papers on how this architecture spread afterward, so the intervening history is omitted. However, the fact that the recent studies below analyze attention inside large language models and video generation models shows that the architecture is still used as a basic building block.

Where it is heading: measuring and fixing attention's quirks

The first line of work concerns problems that appear in long context windows. The attention sink study paper points out that while today's language models advertise context windows of 1 million tokens, two quirks keep them from using the full window. One is the 'attention sink', where attention heads with nothing worth reading dump their leftover share onto the first token. The other is that whether the model finds a fact depends on where in the context the fact sits. The paper cites earlier work reporting that gated attention reduced attention to the first token from 46.7% to 4.8%, then built a measurement tool, SinkProbe, and applied it to four small models that differ only in how they mix tokens. It found that sinks are created by the training objective, not the architecture; that the effect of gating did not replicate at their scale; and that sink size, activation values and position bias moved independently of each other.

The second line of work concerns side effects of position information. The video diffusion model study paper looked inside text-to-video models for the reason they often produce videos that violate physical laws. It confirmed that motion paths form in the early denoising steps ('shape first, details later') and that specific attention heads drive motion planning. It also found that RoPE (rotary position embedding) weakens spatial attention too quickly, so early candidate positions get locked in too soon at physically implausible spots. To fix this, it proposed a lightweight change that adjusts RoPE's frequencies at each denoising step, and reported that it worked both in experiments applied without extra training and in experiments with training.

The third line of work is theoretical research that aims to change attention's mathematical foundation. The Riemannian attention study paper notes that every Transformer-based language model computes attention scores with an ordinary dot product (the Euclidean inner product). Citing prior work, the paper describes the problem that stacking only attention causes the diversity (rank) of representations to collapse rapidly as depth increases. It then proposes a theoretical framework in which each token uses a learned 'way of measuring distance' (a Riemannian metric), along with an architecture specification called the Fiber Bundle Transformer. It states, however, that this is not a proof that rank collapse is prevented, and leaves the core problem as an open question.

Limitations and open issues

First, because self-attention has every token look at every token, the number of token-to-token relationships grows with the square of the sentence length (a general property of the computation). This is the price of the 'constant number of operations regardless of distance' that the original paper emphasized. As context windows grow to the scale of 1 million tokens, a large window does not mean the information in it is used evenly, as the attention sink study paper points out.

Second, it is risky to read attention weights directly as 'what the model considered important.' The attention sink study paper dealt with heads that have nothing to read pouring attention onto the first token without meaning. In other words, a high attention value does not necessarily mean high importance. This is also why interpretability research measures the actual contribution of each attention head separately. The video diffusion model study paper likewise identified the heads responsible for motion by examining both attention patterns and each head's causal contribution.

Third, reports of a fix need to be checked separately to see whether they hold under other conditions. One example is that, in the same study paper, the effect of gated attention did not replicate at small scale. Proposals to change the architecture at its foundation, such as the Riemannian attention study paper, are still at the theoretical stage, and it has not been verified whether they bring improvements in real large models.

Further study

The first thing to read is 'Attention Is All You Need' paper. Check how it viewed the limits of RNNs and CNNs in Section 1, Introduction, and Section 2, Background, then follow the layer structure and masking of the encoder and decoder in Section 3, Model Architecture, together with Figure 1, to revisit this note's 'How it works' section in the original. If the equations are unfamiliar, it is enough to grasp the big picture of each sub-layer: 'mix with attention, refine with the feed-forward network, and preserve the original values with the residual connection.'

After that, choose according to your interests. If you are interested in how language models handle long contexts, read the attention sink study paper. The way it defines its metrics (sink size, recall by position and so on) is also useful for practical evaluation. If you are interested in position information and video generation, the video diffusion model study paper shows concretely how RoPE affects attention. If you are curious about the mathematical properties of attention, read the Riemannian attention study paper, but distinguish the parts the authors themselves mark as 'structural observations, not proofs.'