← Deep dives

Deep dives · 2026-11-13 · Advanced · By bombong

Linear Attention: Reading Long Contexts with a Fixed-Size State

Instead of an ever-growing KV cache, linear attention compresses past context into a single fixed-size state. This saves memory at the cost of weaker exact lookup. Recent research focuses on reducing this weakness by refining the rules for writing to, erasing from and reading the state, and on handling the state to suit hardware and serving environments.

Contents
Key points
  • Linear attention replaces softmax attention's growing KV cache with a fixed-size state, making training compute scale linearly with context length and keeping decoding memory constant.
  • The hard part is not only deciding what to forget. The key is updating compressed memory without scrambling associations already stored, and the delta rule and gates address this problem.
  • Moving from DeltaNet through Gated DeltaNet and KDA to Gated DeltaNet-2, decay became finer-grained per channel, and the amounts erased and written came to be controlled separately.
  • A fixed-size state is weak at pinpointing specific information in the context. So real models often use hybrid architectures that mix in softmax layers, and research continues on fixing the read step or accessing the past chunk by chunk.
  • In serving, the state's storage format and precision drive performance. Low-rank compression improves memory and speed, but mishandling bf16 rounding makes reasoning models' answers longer and lowers accuracy.

Why It Was Needed: A KV Cache That Only Grows

Softmax attention, the core of the transformer, scans the entire context for each token to pull out the information it needs. It is powerful, but its cost grows with the square of the context length, so both training and inference are constrained for long contexts; this is the starting point of a comparative study of linear attention paper. During generation, the keys and values of earlier tokens are stacked in the KV cache (memory that stores already-computed intermediate results for reuse). But this cache grows without limit as the context gets longer.

Linear attention replaces this unbounded cache with a fixed-size recurrent state. In the words of Gated DeltaNet-2 paper, this makes token mixing linear-time and decoding memory constant. HLA-WM paper, a study on video world models, draws the same contrast: softmax attention keeps the entire generation history in a growing KV cache, while recurrent linear attention compresses the history into a fixed-size state and uses far less memory.

The cost is clear too. The CCQ study paper points out that linear attention consistently lags on tasks that retrieve information from within the context and on long-context tasks. So the questions in this field come down to: 'What goes into a fixed-size container, how, what gets erased, and how is it read out?'

Core Idea: Rewriting a Single Whiteboard

Start with an analogy. Softmax attention stacks meeting minutes in a filing cabinet without discarding a single page, and searches the whole cabinet whenever a question comes. Linear attention uses just one whiteboard of fixed size. Each time new content arrives, it rewrites the board, and when a question comes, it looks only at the board. The cabinet keeps growing, but the board always stays the same size.

Here the state is a matrix holding an associative memory: put in a key and a value comes out. Gated DeltaNet-2 paper explains this as updating fast weights. Each time a token arrives, this small weight matrix is adjusted slightly, and multiplying a query by this matrix reads out the related value.

The same paper sums up the difficulty this way: the problem is not just 'what to forget' but 'how to revise compressed memory without disturbing associations already stored.' If you just write new content over the whiteboard, the writing gets jumbled. So delta-rule models first subtract the existing content read by the current key before writing the new value. In effect, they erase the old value bound to that key and write the new value in its place.

It is also worth noting what this design trades. Linear attention gives up 'the ability to look at the original verbatim at any time' and gains 'constant memory regardless of context length.' Most later research is about how much of what was lost in this trade can be recovered.

Softmax attention vs. linear attention

Softmax attentionStack everything in a filing cabinet

  • Can revisit the entire context
  • KV cache grows with context length
  • Cost grows with the square of length

Linear attentionRewrite a whiteboard

  • State size is fixed
  • Decoding memory is constant
  • Weak at exact retrieval and long-range memory

공통 Both are layers that mix information from past tokens into the current token

A side-by-side view of what each approach gains and gives up.

How It Works: Decay, Erase, Write, Read

At inference time, a Gated Delta-family layer updates the state for each token in the following order. First, it computes the key, value and query from the current token. Second, decay blurs the whole state slightly so that old memories fade gradually. Gated DeltaNet adjusts the degree of forgetting based on the input, and Kimi Delta Attention (KDA) handles forgetting more precisely by having each channel forget at a different rate paper. Third, it erases the existing prediction read by the current key. Fourth, it writes the new value bound to the key. Fifth, it reads the state with the query to produce the output.

Gated DeltaNet-2 paper pointed out a limitation of earlier methods in the third and fourth steps: a single scalar gate jointly decides how much old content to erase on the key side and how much new content to write on the value side. So it uses separate per-channel erase and write gates. If the two gates collapse into the same scalar, it becomes KDA; if decay also collapses into a single value, it becomes Gated DeltaNet. In other words, it is a generalization that contains the two earlier models as special cases.

Training is different. Updating the state one token at a time in sequence fails to exploit GPU parallelism. So the sequence is split into chunks (groups of fixed length). Computation within a chunk is handled all at once with matrix multiplications, and only the state is passed between chunks. Gated DeltaNet-2 preserved this parallel training by presenting a chunkwise algorithm that absorbs per-channel decay into the erase-side coefficients, together with backpropagation that accounts for the gates.

Real models are often built as hybrids that mix in some softmax attention layers rather than stacking only linear layers. In the comparative study paper, hybrid configurations generally lowered loss, but at the cost of reduced training throughput.

Per-token processing in a Gated Delta layerNew token → Decay; Decay → Erase; Erase → Write(Gate separation); Write → Fixed-size state(Update); Fixed-size state → Read; Read → OutputNew tokenGenerate key, value,queryDecayFade old memoriesEraseSubtract old valueread by keyWriteBind new value to keyFixed-size stateAssociative memory matrixReadLook up by queryOutputGate separationUpdate
Per-token processing in a Gated Delta layer The top row is the order in which a new token updates the state; the bottom row is the process of reading the updated state with a query to produce the output. The state size stays the same regardless of context length.

How It Developed: From Write Rules to Reading and Hardware

The first line of work refines the write rule, going from DeltaNet through Gated DeltaNet and KDA to Gated DeltaNet-2 paper. Gated DeltaNet-2 reported the best overall results in language modeling, commonsense reasoning and other tasks when 1.3B-parameter models trained on 100 billion FineWeb-Edu tokens were compared with Mamba-2, Gated DeltaNet, KDA and Mamba-3 variants. The comparative study paper put these four architectures into a single recurrent-memory notation, showing where they differ in expressiveness, memory decay, erase/write control, training throughput and implementation difficulty. In its setting of 350M-parameter models trained on 15 billion tokens, KDA trained with the Muon optimizer had the lowest final validation loss, and a pure Gated DeltaNet configuration trained with AdamW had the highest training throughput.

The second line of work is on the read side. CCQ paper points out that most earlier improvements focused on the write side, leaving the read step unchanged. Because all past keys are added into the output, the information actually needed gets diluted, buried under a large mass of other vectors. This study keeps updating the key covariance recurrently in the same way as the state, and shrinks the query along directions where memory is heavily concentrated before reading. When added to GLA and Gated DeltaNet, it improved S-NIAH retrieval, length extrapolation from 4K to 20K, and LongBench accuracy. HLA paper added routing to Gated DeltaNet that decides, for each query, which past chunks to draw on and how much. On Qwen3.5 0.8B–9B models, LongBench-V2 improved by up to 5.57 percentage points and RULER by up to 3.97 points. In experiments training a 1.3B model from scratch, the RULER gain grew from 0.83 points at 4K context to 4.22 points at 32K context. HLA-WM paper found severe long-range forgetting in Gated DeltaNet, where information from distant scenes is gradually weakened as later updates push it out. To address this, it added a training-free method that uses camera geometry to find relevant past chunks and reassemble a per-query state. With a 60-second context, it used 12 times less past-state memory than a full KV cache while raising PSNR by 0.74 dB and cutting rotation error by 28.5%.

The third line of work is about actually building and running these models. Taylor-Calibrate paper addresses converting a pretrained transformer into a hybrid Gated DeltaNet. It argues that simply copying the attention projections leaves the behavior of the decay, write and output gates undetermined, so the converted model starts from a poor state. This study initialized these values using statistics from the teacher attention and reached the same recovery target with 4.9–9.2 times fewer training tokens than naive conversion. Systolic Linear Attention paper showed that porting GPU-oriented Gated Delta layers directly to TPUs leaves the tiles of the 256×256 matrix units about half empty and creates bottlenecks in vector operations. After redesigning the kernel by widening the key dimension to 256 and splitting the decay gate into four groups of 64 channels, it ran 1.6–2.5 times faster than the open-source Tokamax KDA kernel on TPU v7x. In MoE pretraining at a scale of 2,048 chips, it trained 11–24% faster than KDA while maintaining similar perplexity.

Where It Is Heading: Refining Gates and Handling the State

On the architecture side, research has examined where and how many parameters to spend on gates. Muir paper showed that state updates are much less sensitive to erasing than to writing. Key energy concentrates in a few channels, while value writes affect every channel, so the erase gate can be built at low rank. The authors also found that slowly decaying state rows grow larger and dominate the read output, so they added a read gate that reweights each row. The saved parameters went into a wider feed-forward layer. At scales from 166M to 1.34B, perplexity was 1.8–2.0% lower than Gated DeltaNet-2.

On the serving side, the state itself is compressed. SinkState paper confirmed that queries, keys and values in linear layers concentrate in a stable low-dimensional subspace, and called this the "linear attention sink." Each state is therefore effectively low-rank, yet existing serving systems keep it as a dense matrix and read and rewrite all of it at every decoding step. This memory traffic grows with batch size and accounts for most of the decoding cost. By keeping the state in low-rank form, SinkState cut peak state memory by up to 2.6x and sped up decoding by up to 1.58x across four hybrid models from 4B to 120B.

Precision problems also surfaced. Lost in Rounding paper showed that when Mamba-2 or Gated DeltaNet layer states are stored in bf16 with round-to-nearest, five of six hybrid reasoning models reason for longer. On Qwen3.5-9B, the number of problems that hit the 32K-token cap rose from 118 to 223, and accuracy fell from 85.6% to 70.8%. With stochastic rounding at the same bf16 width, accuracy recovered to 87.6%. The authors concluded that the failure comes mainly from rounding bias, not from insufficient representational resolution. Because the state accumulates and carries over at every step, the storage format becomes part of model quality.

Notable numbers from recent research

Up to 2.6xPeak state memory reduction (SinkState)
Up to 1.58xDecoding speedup (SinkState)
1.8~2.0%Perplexity reduction vs. GDN-2 (Muir)
85.6%→70.8%Accuracy with bf16 rounding (Qwen3.5-9B)
Values reported by SinkState, Muir and Lost in Rounding under their own experimental conditions.

Limitations and open issues: when to use it and when to avoid it

The biggest limitation is still exact retrieval. The read dilution problem pointed out by CCQ paper and the long-range forgetting confirmed by HLA-WM paper stem from structural properties of a fixed-size state. For tasks that require finding a specific phrase verbatim in a long document, it is safer to first consider hybrids that mix in softmax layers or read-augmentation techniques rather than models built only from linear layers. However, as the comparative study paper showed, hybrids lower loss at the cost of throughput.

Care is also needed when reading the numbers. The comparative study paper itself states that it measured only training throughput and iteration time and provides no inference speed benchmarks. Experiments in many architecture papers were also mostly run at around 1B parameters. These results should not be read as "the service will get that much faster."

There are three common mistakes in operation. First, treating state precision as just a memory-saving setting and using bf16 rounding as is. As the results of Lost in Rounding paper show, reasoning length and accuracy can both become unstable. Second, porting kernels tuned for GPUs directly to other accelerators. Systolic Linear Attention paper showed that matrix tiles end up about half empty on TPUs. Third, simply copying the weights of an existing transformer and starting distillation right away. According to Taylor-Calibrate paper, doing so spends a large share of training tokens on correcting the initialization.

In short, linear attention fits well when memory is the bottleneck in long-context generation, or when there are many concurrent requests and you want to cap memory at a fixed amount per request. Conversely, if retrieving source text from the context exactly is the key quality metric, check the hybrid ratio and retrieval task evaluations first.

Checklist before adopting linear attention

Use case

  • Is exact source-text retrieval the key quality metric?
  • Have you evaluated hybrid ratios and retrieval tasks?
  • Is memory actually the bottleneck?

Interpreting numbers

  • Did you avoid reading training throughput as inference speed?
  • Is the experimental scale similar to your model?

Operations

  • Have you checked state storage precision and rounding method?
  • Are kernels tuned for your accelerator?
  • Did you initialize gates separately during conversion?
The limitations and common mistakes covered in the text, turned into check items.

Further reading

To get the overall map first, start with the linear attention comparative study paper. It compares DeltaNet, Gated DeltaNet, KDA and Gated DeltaNet-2 in a single notation, which makes the equations in later papers much easier to read. Next, look at Gated DeltaNet-2 paper for why it separates erasing from writing and for its chunkwise training algorithm.

For weaknesses and their fixes, read CCQ paper and HLA paper as a pair. The former fixes the read step, the latter per-query chunk access. As an application, HLA-WM paper shows how long-range forgetting appears in video world models.

If your interest is practical, look at Taylor-Calibrate paper for converting existing models, Systolic Linear Attention paper for hardware-specific design, and SinkState paper and Lost in Rounding paper for state management in serving. For the latest direction in architecture design, continue with Muir paper.

Related terms

Attention Transformer KV cache State space model RNN Context window Precision (FP32, BF16, FP8) Distillation

References

Foundational papers

Recent research

Written by AI from the listed references and reviewed. Please check the original sources for details.