Study notes · 2026-10-10 · Advanced

Looped transformers: running the same layers multiple times to think more deeply

A looped transformer increases computational depth by repeating the same transformer block multiple times instead of adding parameters. However, increasing the number of loops makes training unstable and increases memory use. Most recent research focuses on solving these two problems.

Contents

Why it was needed: going deeper without making the model bigger
Core idea: giving the same manuscript to the same editors multiple times
How it works: passing through one block N times
How the field developed: stabilization, memory, mechanism analysis
Where it is heading: running more loops, more cheaply
Limitations and open issues
Further reading

Key points

A looped transformer reuses the same block multiple times, increasing only computational depth while keeping the parameter count and context length unchanged.
The number of loops can also be changed at inference time. So it acts as a knob for choosing the balance between performance and test-time compute.
Adding loops easily causes the residual to blow up or gradients to fluctuate, collapsing training. For this reason, rules for controlling residual magnitude and loop-specific residual connections are being studied.
Because a separate KV cache is usually used for each loop, memory grows in proportion to the number of loops. Research on sharing or compressing the cache is reducing this burden.
Strengths have been reported in length generalization and state tracking. However, there are also analyses showing that errors accumulate as the repetition gets deeper and that the ability to self-correct is limited.

Why it was needed: going deeper without making the model bigger

Usually, improving model performance means making the model bigger. This is done by stacking more layers and adding parameters (the numbers a model learns), which increases storage and training costs accordingly. The fully looped transformer study paper describes looped transformers as an alternative to this formula. Because they reuse the same transformer block repeatedly, they gain performance by doing more computation while keeping the parameter count and context length unchanged.

Another advantage is that the number of loops can be changed at inference time. By running more loops on hard problems and fewer on easy ones, you can choose the balance between performance and test-time compute (the amount of computation used to produce an answer) paper.

In a standard transformer, the depth (number of layers) applied to each token is fixed. The recurrent looped transformer study paper points this out. In state-tracking problems, where the state must be updated with every new input, the depth stays the same even as the sequence grows longer, which creates a limit. The question 'can computational depth be increased flexibly?' is driving research on looped architectures.

Core idea: giving the same manuscript to the same editors multiple times

As an analogy, a standard transformer is like having 24 editors each revise a manuscript once, one after another. Each editor has their own skills (parameters), so the more people you add, the higher the labor cost (parameter count). A looped transformer forms a team of a few editors and has this team pass over the same manuscript multiple times. The headcount stays the same, but the number of revision passes, that is, the 'unrolled depth', increases.

DeepLoop paper summarizes this as 'applying a compressed set of physical blocks over multiple rounds to increase unrolled depth without increasing stored parameters.' Several studies view this as a new scaling axis called 'recurrent depth.' This means that besides model size and data volume, how many times the same block is run becomes another performance knob.

However, when the same editors revise the same manuscript many times, they sometimes ruin parts that were previously fixed well. Much of the research on looped transformers is about 'how to keep the state from collapsing even as repetitions increase.'

Standard transformer vs. looped transformer

Standard transformerDifferent weights per layer

More depth means more parameters
Fixed depth per token
Well-established training rules

Looped transformerRepeats the same block

Adds depth without adding parameters
Loop count adjustable at inference
More loops bring training instability and memory growth

공통 Both use transformer blocks made of attention and residual connections

They gain depth in different ways. The looped approach increases depth through repetition count instead of parameters.

How it works: passing through one block N times

The process can be broken into steps. 1) Input tokens are converted into embeddings (numeric vectors). 2) This representation (the model's intermediate internal representation) is passed through a shared block made of a few layers. 3) The output is fed back into the same block. This repeats for a set number of loops. 4) The representation from the final loop is used to predict the next token. The attention and residual connections (paths that add a change to the input and pass it on) inside the block are the same as in a standard transformer. The only difference is that the same weights are used multiple times.

This repetition creates two burdens. The first is training. According to DeepLoop paper, in a standard transformer each layer's parameters are updated separately. In a looped structure, one shared update collects gradients from multiple visits, and in the next forward pass those visits all read the same update again. So the rule for setting residual scale has to change.

The second is memory. Ouro, a representative looped language model, keeps a standard KV cache (memory that stores the keys and values of previous tokens) for each loop. MELT paper points out that because of this, memory grows linearly with reasoning depth. The problem has also shown up in real deployment. According to an analysis of Nanbeige4.2-3B paper, this model passes one group of layers twice, and because of the layer reuse its peak attention memory effectively doubles.

How a looped transformer processes input The input passes repeatedly through the shared block a set number of times before output. The dotted line shows the memory burden of the KV cache accumulating with each loop.

How the field developed: stabilization, memory, mechanism analysis

The foundational studies fall into three broad lines. The first line is training stabilization. The fully looped transformer paper diagnosed the causes of instability as loops increase as gradient oscillation and residual explosion. It then proposed two fixes that add no parameters: a structure that distributes the signal between loops across all layers, and attention injection that reuses the existing attention block. As a result, it trained stably up to 12 loops, where other looped models break down. Even in settings where looped models do not break down, it raised average downstream task performance by up to 13.2%. DeepLoop paper adapted the DeepNorm rule for setting residual scale to the looped structure. At GPT-2 small and medium scale, validation loss and accuracy improved when the same block was revisited. The study concludes that 'stable recurrent depth requires a rule that accounts for the number of parameter visits, not the nominal number of layers.'

The second line is memory and deployment. Instead of keeping as many KV caches per layer as there are loops, MELT paper keeps one cache per layer shared across loops and updates it with a learnable gate. Starting from pretrained Ouro parameters, it was trained in two stages that include distillation (knowledge distillation). As a result, it used memory similar to a standard LLM of comparable size while performing better. The Nanbeige4.2-3B analysis paper found five bugs that kept the public checkpoint from running out of the box on Apple Silicon. It also used chunked prefill to increase the usable context in 32GiB of shared memory by 2.7x. The training-free looped transformer paper repeats a middle group of layers of an already trained model only at inference time. Simply reapplying the block usually hurts performance. So it split one large update into several damped small steps, and raised Qwen3-4B-Instruct's MMLU-Pro score by 2.64 percentage points.

The third line is mechanism analysis. The recurrent looped transformer paper splits layers into a parallel encoder and a recurrent decoder, and feeds the decoder state of the previous token back into the next token. Trained on up to 40 bits, it solved parity (the task of determining whether the number of 1s is odd or even) up to 256 bits with 100% accuracy. Under the same conditions, a standard transformer stayed at chance level. The length generalization mechanism study paper looked inside looped models to see how they solve reasoning chains longer than those seen in training. It found a shared principle: the recurrent state holds not only task content but also whether that content is 'ready to be used in the next computation.' The energy-entropy regularization study paper trained a single-head looped model, which is hard to train because of its rugged loss landscape, with a method that treats it like a physical flow. A model with dimension 8 solved an induction head task with a length of 1,000 tokens.

Where it is heading: running more loops, more cheaply

Recent research aims to 'make models hold up even when run for far more loops.' InfiLoop paper started from the observation that adding loops can actually lower reasoning accuracy. Noisy state updates overwrite correctly reasoned intermediate results and can even undo answers already solved. To prevent this, it built a loop-specific residual connection that learns what to keep from past computation and how much of a new update to accept. With a small 7M-parameter model, it reached 97.9% accuracy on Sudoku-Extreme, and performance kept improving beyond 20,000 effective steps.

Other work extends loops to large MoE (mixture of experts) models. According to LOOM paper, adding loops used to bring quickly diminishing gains or even make things worse, so prior work mostly stayed at 2 loops. It identified two causes: the variance of hidden states growing with each iteration, and 'expert selection collapse,' where the router picks the same experts in every loop. Using methods such as scaling residual updates, re-injecting input embeddings at each loop and per-loop routers, it scaled stably to 9–12 loops in 100M–1.7B models.

Another direction reduces inference cost. LoopCD paper uses the fact that predictions from earlier loops are 'weaker predictions with less computation.' It picks the next token by contrasting them with the final loop's prediction, with no additional training needed. It raised Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33%. Even with the number of loops cut in half, it matched or beat the full-depth baseline, reducing forward-pass FLOPs by 22.5–48.2%.

Notable numbers from recent research

20,000+ stepsInfiLoop: performance improves with more recurrence

9–12 loopsLOOM: stable loop count for MoE

22.5~48.2%LoopCD: forward-pass FLOPs reduction

Each number comes from one of the recent studies introduced in the text.

Limitations and open issues

The biggest issue is instability as recurrence gets deeper. Several stabilization techniques have appeared, but the reported stable range is generally around 9–12 loops paper paper. The length generalization mechanism study paper reported that in both loop settings it examined, the internal mechanisms become less reliable as depth grows. The points where the state is read and the progress signals become blurred, and state propagation becomes less reliable. The ability to self-correct is also limited, so a local error, once made, persists and keeps accumulating.

The second issue is that the advantage of 'fewer parameters' does not reduce the total cost. Computation grows with the number of loops, and the KV cache grows along with it unless it is handled separately paper. The Nanbeige4.2-3B analysis paper shows the difficulties of real deployment. Agent tasks could be evaluated only after the bugs and memory problems were fixed. Even then, completion rates on some MCPMark tasks were at most 30%, and on BFCL the model failed most tests that require using several tools together.

Third, the gains depend on the task. In the recurrent looped transformer paper, 64-bit parity accuracy stayed at 99% even when feedback was updated only once every four tokens. In contrast, accuracy on a permutation tracking task dropped from 100% to 20%. In the training-free loop paper, simply repeating the block usually made performance worse. The simple expectation that 'adding loops makes things better' does not hold.

References

Foundational papers

Recent research

Written by AI from the listed references and reviewed. Please check the original sources for details.