Paper review · November 18, 2026
K2-Horizon: six open-source language models released with full data, checkpoints and code
K2-Horizon: A Frontier Fleet of Open Source Language Models
The authors released six models, from 0.9B to a 375B-A23B MoE, together with training data, intermediate checkpoints and training code, and achieved performance competitive with open models of similar size on agent and coding benchmarks.
Problem
Many open-weight models exist, but the data, intermediate states and decisions behind them are rarely released. This makes it hard to study, under controlled conditions, how scale or sparsity (MoE) affects performance.
Method
Following LLM360's definition of open source, the authors released the final weights of six models, 380 checkpoints from different training stages, training logs, 21.6TB of training data, the data generation pipeline, the training framework (xLLM), the RL stack (RL360) and the evaluation tools (Eval360). On the architecture side, Grouped RMSNorm improves training stability. They also introduced MoVA, which applies MoE to the Value Projection in attention, and a Softplus Attention Gate. About 10T tokens of synthetic pretraining data were used, and 4 models were trained on the same 22T-token corpus and recipe. For the 375B model, separate RL expert models were trained for search, knowledge work, tool use, scientific reasoning and instruction following, combined through weight merging and then refined with SFT. For inference speed, the authors also built Uno, a lossless diffusion adapter that does not require a separate Draft model.
Results
- The authors reported that the 375B-A23B model outperformed four contemporary open-weight models (GLM-5.2, MiniMax M3, Nemotron 3 Ultra, Inkling) on Toolathlon-Verified (65.3) and on repository-level code understanding without internet access (SWE-Atlas-QnA 48.4).
- The 36B-A4B model using MoVA matched or came within a few points of a 32B Dense model trained on the same data on every benchmark. Its active parameters are about one-seventh as many.
- The Uno adapter increased decoding speed by up to 2.2x.
- The authors reported that the normalized loss curves of the 4 models trained with the same 22T tokens and recipe (Dense and MoE, differing in scale by about 10x) nearly matched.
Limitations
The authors noted a performance gap on expert-level scientific reasoning such as HLE and CritPt. Comparisons are limited to some open and closed models from a similar period. The provided text is cut off partway, so full per-benchmark figures and the safety evaluation could not be confirmed.
Why it matters
By releasing the data pipeline and intermediate checkpoints, the work lets companies and researchers reproduce the training process or analyze the effects of scale and sparsity through controlled experiments. It also offers options from 0.9B to 375B for different deployment environments.
Key terms
- MoVA(Mixture-of-Value Attention)
- An architecture that places multiple experts in the attention Value computation, with a router selecting only some of them. Only the combined Value is cached, so the KV cache size does not grow.
- Grouped RMSNorm
- A method that splits the feature dimension into groups and normalizes each group separately. It stabilizes training by preventing normalization outputs from becoming too large in wide models.
- Weight merging
- A technique that combines the parameters of models trained separately for different domains into a single model. It keeps most of the performance gains each expert model achieved.
- A23B notation
- In an MoE model, it means 23B active parameters are actually used in computation when processing one token.
Original
Authors: K2 Team, Zhengzhong Liu, Haonan Li, Xuezhe Ma, Alexander Moreno, Subham Sekhar Sahoo, Rupesh Kumar Srivastava, Mikhail Yurochkin, et al. · Posted 2026-11-16 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.