Paper review · November 19, 2026
Multi-Branch Transformer: Arranging blocks as parallel branches to improve both quality and inference speed
Multi-Branch Transformer
The authors rearranged a Transformer's middle blocks into independent parallel branches and shared the KV cache across branches, improving both quality and decoding throughput at the same compute.
Problem
Transformers are usually scaled by widening blocks (width) or stacking more of them (depth). The "branch" direction, which places entire blocks in parallel, has barely been studied.
Method
After shared early layers (prelude), MBT runs K branches with independent parameters in parallel. Branch outputs are merged with RMSNorm, concat and a linear projection, then passed to shared later layers (coda). At the same compute, the authors compared three allocations: Wide-Shallow, Narrow-Deep and Balanced. They also introduced Joint-KV, which combines branch inputs at the same depth into a single KV cache. The nano-vLLM-based inference engine adds branch grouping and head folding. Pretraining experiments were run at four MoE scales from 0.8B to 2.1B (FineWeb-Edu, about 200 TPP).
Results
- With about 1/3 of blocks placed in branches, a 2-branch Wide-Shallow MBT had validation loss 0.025–0.031 lower than Standard at all 4 scales, equivalent to about a 1.23x body compute advantage
- Average accuracy across 9 zero-shot benchmarks, averaged over W1–W4, rose from 54.23% to 56.03% (higher than Standard on all 9)
- 2-, 3- and 4-branch MBT with Joint-KV matched the performance of 1.16–1.26x the training body compute of Standard, while reducing the KV cache by 19–34%
- On a single H200 (64K input, 8 concurrent requests), decoding throughput was up to 51.4% higher than Standard
Limitations
Experiments go up to 2.1B total parameters (about 0.4B active), with a single dataset (FineWeb-Edu) and a 4,096-token context. Whether the same trends hold at larger scale has not been verified. Splitting branches across multiple devices requires all-reduce synchronization at each depth. MBT's learning rate reused a value tuned in one setting.
Why it matters
The paper proposes "parallel branches" as a new scaling axis beyond width and depth. Because it can improve quality at the same training budget while also reducing long-context serving costs (KV cache, number of sequential steps), it is worth considering for model designs sensitive to inference cost.
Key terms
- Prelude / Coda
- The groups of shared layers before and after the branches in MBT. The prelude output goes into each branch, and the merged result goes to the coda.
- Joint-KV
- A method that concatenates all branch inputs at the same depth to build a single KV cache, while keeping queries separate per branch. The KV cache in the branch region shrinks to 1/K.
- Body compute
- The training compute of the model body, excluding embeddings and the LM head. In this paper it is the basis for comparing efficiency across architectures.
Original
Authors: Gaotang Li, Lingjie Chen, Chunyuan Deng, Zhiwei Xu, Hongzheng Chen, Christopher De Sa, Jingrui He, Yingheng Wang, et al. · Posted 2026-11-15 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.