Paper review · October 24, 2026
IQuest-Q1: An open MoE foundation model for agentic CLI systems
IQuest-Q1: Advancing Foundation Capabilities for Agentic CLI Systems
Through a stable pretraining design, staged context extension, and multi-scaffold reinforcement learning and MOPD, the authors built and released a 320B (15B active) MoE agent model.
Problem
The command-line interface (CLI) is the most general-purpose channel through which language models access tools, files and code execution. Building stronger agents therefore requires CLI ability. At the same time, designing training that scales large models stably and efficiently is also a challenge.
Method
It is an MoE model that activates 8 of 256 experts (88 layers, hybrid attention mixing GQA and SWA at 3:1, Query-Aware Sink). Stable SplitNorm, which normalizes only the attention input, is combined with the Muon optimizer and μP so that learning rates found on a small proxy model transfer directly. In the second pretraining stage, the share of code was raised from 40% to 70%, and during mid-training the context was extended 32K→128K→512K (29T tokens in total). Agent trajectories were generated from more than 1 million containerized repository instances. Post-training combined SFT, domain-specific reinforcement learning and multi-teacher on-policy distillation (MOPD) across multiple scaffolds. Finally, models are merged, and one MTP layer is converted into a recursive drafter for speculative decoding.
Results
- Scaling efficiency is 1.93x that of an MoE baseline with the Qwen3-30B-A3B configuration trained on the same data. The authors report that reaching the same training loss required only about 52% of the compute.
- Extrapolating the critical model size (CMS) they propose, the authors estimate a stability margin up to about 572B active parameters and about 12T total parameters.
- The authors state that it performed well on 8 agent and reasoning benchmarks, including DeepSWE v1.1, CyberGym, NL2Repo, JobBench and Humanity's Last Exam (the provided text does not include specific numbers).
Limitations
In the provided text, per-benchmark scores are mentioned only in figures, so quantitative comparisons with other models cannot be checked. One of the key metrics, IQuest-CLIBench, is a private internal benchmark. The CMS figures are estimates extrapolated beyond the measured range. The scaling efficiency comparison is also against a single 30B-class baseline.
Why it matters
It adds another large open-weight MoE model specialized for coding and CLI agents. Training know-how, such as learning rate transfer, a metric for measuring stability margin (CMS) and methods for generating agent trajectories at scale, is disclosed in relatively high detail.
Key terms
- MoE(Mixture of Experts)
- An architecture that selects only a few of several expert FFNs for each token. Total parameters are large, but compute per token stays at the level of the active parameters.
- μP(Maximal Update Parametrization)
- A method for setting initialization and learning rate scaling so that optimal hyperparameters stay the same as model width changes. Values tuned on a small model can be transferred to a large model.
- MOPD
- Multi-teacher On-policy Distillation. A method that merges knowledge from several frozen teacher models with different abilities into one model, using the student's own rollouts.
- Speculative Decoding
- A technique that speeds up inference by having a small drafter propose several tokens ahead, which the main model then verifies in a single pass.
Original
Authors: A. Jian Yang, An Huang, Bo Dan, Chen Mu, Chengran Yang, Chuan Shi, Feng Chang, Hongxin Li, et al. · Posted 2026-10-22 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.