Recurrent depth: thinking deeper by running the same layers multiple times

Why it was needed: why stacking layers is not enough

A standard transformer stacks a fixed number of layers, each with different parameters. The input passes through these layers once and is done, so the "depth of thought" the model can use within a single computation (forward pass) is fixed by the number of layers. The only way to make a model think about a problem longer was to have it write out intermediate reasoning as text, as in CoT (chain-of-thought).

The recurrent-depth transformer study paper frames this limitation in terms of "implicit reasoning." Large language models memorize many facts and rules, but often fail to perform multi-step reasoning that chains them together within a single computation. The study views this as a sign that the model lacks the generalization ability to combine knowledge it already has.

Another problem is that parameter count and computational depth are tied together. Computing more deeply requires stacking more layers, which makes the model larger. Recurrent depth is an attempt to decouple the two. By using the same layers multiple times, compute can be increased while the model size stays the same.

Core idea: run the same block multiple times

The parallel sampler study paper describes recurrent depth as a language model architecture that can increase compute by repeating layers, and notes that in transformers it is also called a Universal or looped transformer (Loop Transformer). The core idea is to apply a single shared block T times in succession, progressively refining the internal representation (the numeric vectors that carry information inside the model).

As an analogy, consider two ways of editing a manuscript. A standard transformer is like a publisher where twelve editors each revise the manuscript once in turn. A recurrent-depth model is like a single skilled editor who rereads and revises the same manuscript several times. There are fewer editors (parameters), but more rereading time (compute) can be spent on a difficult manuscript.

The key point is that the number of iterations can be chosen at run time rather than at training time. Fewer iterations are cheaper and faster; more iterations compute more deeply. For this reason, recurrent depth is seen as a way to increase test-time compute (spending more compute at inference time) inside the model, without generating more text. The two-stage dynamics study paper also describes recurrent-depth transformers as "an architecture that increases test-time compute by repeating latent computation before emitting a token."

How it works

The process can be viewed roughly in four steps. First, input tokens are converted into embeddings. Second, the shared block is applied once to update the internal state. Third, the updated state is fed back into the same block, repeating a fixed number of times or until a stopping condition is met. Fourth, the next token is read out from the final state. Output can also be read from the state at each iteration, so a single model can produce answers at multiple depths.

The two-stage dynamics study paper examined what the iterations do internally. According to the study, updates within a single loop block are small, precise refinements, while larger moves occur between consecutive blocks. As training progressed, the size of each loop step shrank and the step directions became increasingly close to orthogonal to one another; the researchers interpreted this as a sign that the model refines fine-grained structure rather than just pushing in one direction.

When to stop is also a design element. The same study proposed early stopping based on the second-order difference of the step size (the change in the change), and reported that it outperforms KL-divergence criteria and simple first-order difference criteria in performance, stability and time efficiency. On the vision side, RD-ViT paper added ACT (Adaptive Computation Time, a method that uses different amounts of compute at each position); during training the average number of iterations dropped from 2.6 to 1.4, and the model spent more compute on boundary regions.

How the field developed: from understanding principles to speed, safety and applications

The source papers were published between fall 2025 and October 2026 and fall into four broad lines. The first line is understanding the principles. The two-stage dynamics study paper analyzed the geometry of the iterations, and the recurrent-depth transformer study paper ran controlled experiments with models trained from scratch. In that study, standard transformers struggled with both systematic generalization, which combines knowledge never combined during training, and depth extrapolation, which performs 10-step reasoning after training on up to 5 steps, but recurrent-depth transformers succeeded at both. Systematic generalization appeared as three-stage grokking (a phenomenon where generalization suddenly emerges after long training), moving from memorization → in-distribution generalization → systematic generalization, and depth extrapolation could be achieved by increasing the number of iterations at inference time.

The second line is speed. The parallel sampler study paper focused on similarities between recurrent-depth models and diffusion language models and built a sampler that emits a new token at each computation step while refining the states of earlier tokens in parallel through iteration. Applied without additional tuning to an existing 3.5 billion (3.5B) parameter recurrent-depth transformer, it achieved up to a 5x speedup. RAMP paper targeted the fact that repeated computation increases cost in the prefill stage, which processes long inputs. Based on an analysis showing that quantization error introduced in earlier iterations is reduced in subsequent iterations, it computes earlier iterations at low precision and the final iteration at high precision. It maintained near full-precision quality up to 4K length while speeding up the first prefill by up to 1.16x (13.9% lower latency) on NVIDIA H100.

The third line is safety. The SafeBridge study paper pointed out that because output can be read separately at each depth, the output space exposed to attack becomes larger. Jailbreak attack success rates can actually rise at deeper iterations, the same prompt can show different safety behavior at different depths, and attacks targeting one depth transferred to other depths. SFT and preference alignment did not eliminate these differences, so the study proposed depth-specific control and an alignment method that supervises multiple depths together.

The fourth line is applications. RD-ViT paper brought this architecture to medical image segmentation: in a 2D setting using only 10% of the training data, it reached a Dice score of 0.774, ahead of a standard ViT (0.762); in 3D, with MoE added, it achieved 0.812—99.4% of the standard ViT's performance (0.817)—with 3 million (3.0M) parameters, 53% of the parameters. In the clinical text study paper, Ouro-1.4B-Thinking scored higher than Qwen3-1.7B on 11 of 13 datasets, and Ouro-2.6B-Thinking was comparable to Qwen3-8B on most tasks. In contrast, HRM-Text-1B showed mixed results.

Where things are heading

The main trends in recent research are "making more iterations actually produce better results" and "reducing the cost of iterating." PAR paper starts from the observation that increasing the number of iterations does not automatically solve longer problems. When trained only on final answers, how much progress each iteration should make is left undetermined, and the magnitude of the internal state drifts as iterations proceed. PAR trains the prediction at the k-th iteration to match the state after completing the first k operations, and rescales the iteration state back to the magnitude of the first iteration.

As a result, on a permutation state-tracking task (swap-S5), a Qwen2.5-0.5B model trained on up to 32 moves achieved an average accuracy of 95.5% at 512 moves, 16 times the training length. On the same backbone, RLT was reported at 55.7% and explicit CoT at 0.8%. Removing the state-magnitude normalization reduced the extrapolation factor to below 2.2x, and replacing the progress-aligned targets with final-answer supervision reduced it to at most 8x. This means training design is needed to actually realize the potential shown by the recurrent-depth transformer study paper—that more iterations yield deeper reasoning.

On the speed side, LoopAhead paper adapted speculative decoding (a method in which a small draft is produced first and then verified by a large model) to looped architectures. It drafts multiple future tokens from early-iteration states, uses signals from deeper iterations to decide which draft branches to continue, and leaves the final decision to the full-depth model. It reported speedups of up to 7.82x over autoregressive decoding while preserving the original model's output distribution. Assigning different roles to different iteration depths reflects a trend of treating depth not as a simple "count" but as a resource that can be designed.

Limitations and open issues

The most practical issue is safety. The iteration-reduction study paper examined four public Ouro checkpoints and Huginn-0125. Even when iterations were cut to half the trained depth, Ouro-1.4B-Thinking's accuracy on GSM8K math problems went from 0.775 to 0.692, retaining 89% of the original. Under the same conditions, however, the rate of assisting with harmful requests jumped 5.3x, from 0.038 to 0.205. A deployment decision to cut iterations to save cost erodes safety the most.

Interestingly, the information itself did not disappear. Even from the internal state after a single iteration, a simple linear classifier could read whether a request was harmful with a ROC AUC of 0.967 (0.986 at full depth). What disappeared was the refusal that should follow. Removing a single direction from the internal state of the full-depth Ouro-1.4B dropped the refusal rate on harmful prompts from 0.950 to 0.130. The model's existing early-stopping rule did not spend more iterations on harmful requests, choosing depth per request did not recover the loss, and only changing training was reported to be effective. This points in the same direction as the conclusion of the SafeBridge study paper.

Cost is not free either. RAMP paper points out that iteration uses correspondingly more compute, and the burden is especially heavy in the prefill of long inputs because the entire input is processed repeatedly. Acceleration techniques are emerging, but most are partial solutions targeting specific stages (prefill, decoding).

Finally, the scope of the evidence needs to be distinguished. The depth extrapolation results come from controlled experiments with models trained from scratch or from tasks with clear structure, such as state tracking. The clinical text study paper itself states that more controlled experiments are needed to isolate the effect of the architecture itself. RD-ViT paper's results also come from a single cardiac MRI benchmark and should not be read as real-world clinical performance.

Further reading

To grasp the concept first, start with the recurrent-depth transformer study paper. It uses controlled experiments to show how it differs from a standard transformer and what becomes possible with more iterations. Then the two-stage dynamics study paper shows what shape the iterations take internally and when to stop, which gives a feel for how it works.

If you build inference systems, read the parallel sampler study paper, RAMP paper and LoopAhead paper together. All three share the approach of exploiting the structural property of "repeating the same block" for acceleration, which makes them good to read side by side.

If you are considering deployment, the SafeBridge study paper and the iteration-reduction study paper are essential reading. They show with concrete numbers that changing the number of iterations as a runtime setting is a change that requires redoing safety evaluation. If you are interested in training design, PAR paper is a good example of what to teach at each iteration.