Paper review · October 9, 2026
STEPQuant: When and Where Error Matters in Delta-rule Recurrent State Quantization
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
By quantizing the recurrent state of linear attention models with error lifetime and location in mind, the method kept FP32-level accuracy at 6 bits while cutting serving memory by up to 68.7%.
Problem
Linear attention uses a fixed-size recurrent state instead of a KV cache. However, a separate state must be kept for each concurrent request, so state memory grows as concurrent requests increase. The authors report that when serving Qwen with SGLang, the FP32 state pool exceeds BF16 weight memory at 70 concurrent requests. Naively applying uniform quantization to this state propagates error into the next state at every decoding step. As a result, accuracy drops sharply at 4 and 6 bits, and a gap remains even at 8 bits.
Method
STEPQuant handles error separately along the time axis and the spatial axis. (1) Lifetime-aware Bit Allocation: gate retention is used to estimate the memory lifetime of each state unit (heads for Qwen, key rows for KDA). Bits are then allocated in a mixed fashion to minimize 'quantization error × lifetime weight.' Some of the riskiest units are kept as FP16 pivots. This allocation is set once during offline calibration and kept fixed for all requests. (2) Key-Row-Aware Dual-axis Fitting: in the state matrix, outliers appear along both key rows and value columns. So separate row scales and column scales are used. Row scales reflect both the magnitude of the row and a score for that row's influence on output error. Column scales are fit to minimize reconstruction error that gives more weight to high-influence rows. Finally, a GPU kernel that fuses state restoration, the Delta update and readout was integrated into SGLang.
Results
- Average accuracy of Qwen3.8-27B on 7 long-form generation reasoning benchmarks (BF16 weights): FP32 state 80.60, STEPQuant 6-bit 80.59, 4-bit 80.51. Uniform INT8 reached only 71.86, INT6 45.04 and INT4 12.73.
- The authors report that 4-bit STEPQuant outperforms uniform INT8 on both models. On Qwen it came close to FP32, while on Kimi-Linear-48B-A3B a small gap remained.
- 6-bit STEPQuant compressed recurrent state memory 5.03 times on Qwen and 5.08 times on Kimi. Total serving memory including model weights fell by 68.7% and 53.7%, respectively.
- The optimized kernel made state updates up to 2.91 times faster.
- Across Qwen's 2,304 recurrent heads, the Spearman correlation between gate half-life and accumulated state error was about 0.80. This supports the assumption that longer-lived states accumulate larger error.
Limitations
Evaluation is limited to two gated Delta-rule hybrid models (Qwen3.8-27B and Kimi-Linear) on A800 GPUs. Kimi's 4-bit setting still shows a gap relative to FP32. Bit allocation, FP16 pivots and row influence scores are set offline using 32 WikiText-2 segments (2,048 tokens each) and then fixed for all requests. So this experiment alone cannot show whether performance holds when the input distribution differs substantially. Also, '6-bit' is a nominal budget that includes the FP16 pivots.
Why it matters
Linear attention and hybrid models are memory-efficient for long contexts. But in concurrent serving, the recurrent state becomes a new memory bottleneck. This work analyzes why state quantization fails and also presents a practical solution integrated into SGLang. It is directly useful for teams trying to reduce the cost of serving hybrid models.
Key terms
- Recurrent State
- A fixed-size matrix in which linear attention holds a summary of past token information. Its size stays the same as the sequence grows, but one must be kept for each request.
- Gated Delta Rule
- A state update rule that decays the previous state with a retention gate and writes a new value by correcting error along the current key direction. Gated DeltaNet and KDA use this approach.
- Mixed-precision quantization
- A quantization method that gives more bits to sensitive parts of a model and fewer bits to less sensitive parts. The goal is to preserve more accuracy within the same average bit budget.
- Post-training quantization (PTQ)
- A method that sets quantization parameters using only a small amount of calibration data, without retraining.
Original
Authors: Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Zhenan Sun, et al. · Posted 2026-09-29 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.