TokenRouter: an efficient serving system for token-level LLM routing

Problem

Token-level routing, which switches the model that handles each token, has been reported to offer cost and quality benefits at the algorithm level. However, existing serving systems such as SGLang and vLLM assume a single LLM, which leads to step desynchronization across models, delayed batch entry and high implementation complexity.

Method

Developers describe only how a single request moves between models, using three functions: route, send and receive. The runtime launches a subserver for each model and runs them asynchronously. A tri-loop structure, which adds an inter-model loop to the two existing loops, and handoff-resume using a pending state reduce the cost of switching between models. Each subserver uses a delayed batching scheduler that starts a batch once requests for the same model accumulate up to a threshold B, and the optimal B is found with a throughput model based on a discrete-time Markov chain. The system is implemented on top of SGLang.

Results

• Across five routing algorithms (CITER, R2R, R-Stitch, Co-LLM and ME) and multiple workload and model combinations, the authors report decoding throughput 2.01 to 64.15 times higher than existing systems.
• At 8 concurrent requests, engineering optimizations alone improved throughput 1.71 times over the official R2R implementation, and the full TokenRouter improved it 2.76 times.
• Under stricter SLO conditions, it achieved 18.58 times higher throughput than R2R.
• The main experiments used a Qwen3-0.6B and Qwen3-32B combination (with Qwen3-8B added for ME).

Limitations

The provided text was truncated, so detailed experimental results and the effect on quality could not be confirmed. The evaluated models are limited to Qwen3-family combinations. Delayed batching can increase waiting time for requests that arrive first, so the balance between throughput and per-request latency may vary by workload.

Why it matters

Token-level routing can cut costs by handling most tokens with a small model and passing only difficult tokens to a large model. It is not usable unless actual serving speed keeps up, and this work addresses that system-side bottleneck. It also provides an interface that can directly replace a single LLM server.

Key terms

Token-level routing

An approach that decides, for each token during response generation, which model produces the next token. It allows finer control over cost and quality than query-level routing.

Step Desynchronization

The problem where each model has a different latency per decoding step, so when they are grouped into one batch, faster models sit idle waiting for the slowest model.

Delayed Batching

A scheduling method that waits briefly until a certain number of requests for the same model have gathered and then runs them together, reducing average batch-entry latency.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang · Posted 2026-10-08 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.