Paper review · November 11, 2026
Flide: Accelerating decision model inference with exact state forking
Flide: Accelerating Decision Inference with Exact State Forking
The authors built an inference engine that computes a shared context and multiple questions together in a single flat token stream while keeping each question mathematically identical to running it independently, achieving responses up to 9.5x faster than vLLM and SGLang.
Problem
Decision models such as Jev ask multiple questions about a single long context and output a probability for each option. However, generation-oriented engines such as vLLM and SGLang process each question as a separate sequence and recompute the shared context. In hybrid models, the prefix cache is restored only at block boundaries, which causes recomputation. Multi-item scoring distorts output probabilities because of separator tokens, and with dynamic batching the order of floating-point summation changes, so decisions can vary even for the same request.
Method
The engine stores the logical tree structure of requests (shared state = trunk, questions = branches) as metadata in GPU memory and runs all requests in a round as a single unpadded token stream. ForkAttention masks full attention layers so that each question sees only its request's trunk and its own branch. Gated DeltaNet layers are called twice: first to compute the recurrent state for the state segment, then so each question continues computing from that state via indexing. Because the round layout is expressed purely as data, a small number of pre-captured CUDA graphs handle a wide range of request combinations. When agent history grows, computation resumes from a previous checkpoint, and an optional batch-invariant mode gives bit-for-bit identical results.
Results
- The authors report that it answered questions up to 9.5x faster than vLLM and SGLang on public hybrid models and was also the fastest on a dedicated decision model.
- They report that when multiple questions are attached to a single shared state, it had the highest throughput at every concurrency level.
- Its Total Variation Distance from the float32 reference distribution was the smallest among the compared engines. Multi-item scoring, by contrast, gained throughput at the cost of distorted probabilities.
- They prove by induction over layers that, assuming exact arithmetic, each question's hidden state is the same as in independent execution (Proposition 3.1).
Limitations
The correctness guarantee assumes causal models and exact arithmetic; in floating point, rounding-level differences remain. The engine is specialized for decision workloads whose output is option probabilities, so its scope differs from accelerating general text generation. Detailed experimental figures and part of the model list are cut off in the provided text.
Why it matters
Agent systems are making more calls, such as routing, judgment and evaluation, where speed and probability calibration matter. Because it shows a serving design that preserves both speed and probability accuracy, it is useful in practice for teams running threshold-based decisions.
Key terms
- Decision model
- A model that does not generate text but directly outputs a probability for each option in response to multiple questions about a long context. It is used for an agent's fast judgments (System One).
- Gated DeltaNet
- A recurrent layer that accumulates all past tokens into a single fixed-size matrix state. Questions cannot be separated with an attention mask, so separate state forking is needed.
- Prefix caching
- A technique that stores and reuses the KV cache for the leading portion shared by multiple requests. However, each question is still run as a separate sequence.
- CUDA graph
- A feature that records the execution order of GPU kernels in advance and replays it all at once, reducing kernel launch overhead. It has the constraint that kernel shapes are fixed.
Original
Authors: Bobo Li, Jiajia Song, Hao Fei, Wynne Hsu, Mong Li Lee · Posted 2026-10-30 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.