← Key papers

Paper review · November 18, 2026

K2-Horizon: six open-source language models released with full data, checkpoints and code

K2-Horizon: A Frontier Fleet of Open Source Language Models

Language modelsEfficiency & infra Intermediate Upvotes 70 cs.AI Institute of Foundation Models

The authors released six models, from 0.9B to a 375B-A23B MoE, together with training data, intermediate checkpoints and training code, and achieved performance competitive with open models of similar size on agent and coding benchmarks.

Problem

Many open-weight models exist, but the data, intermediate states and decisions behind them are rarely released. This makes it hard to study, under controlled conditions, how scale or sparsity (MoE) affects performance.

Method

Following LLM360's definition of open source, the authors released the final weights of six models, 380 checkpoints from different training stages, training logs, 21.6TB of training data, the data generation pipeline, the training framework (xLLM), the RL stack (RL360) and the evaluation tools (Eval360). On the architecture side, Grouped RMSNorm improves training stability. They also introduced MoVA, which applies MoE to the Value Projection in attention, and a Softplus Attention Gate. About 10T tokens of synthetic pretraining data were used, and 4 models were trained on the same 22T-token corpus and recipe. For the 375B model, separate RL expert models were trained for search, knowledge work, tool use, scientific reasoning and instruction following, combined through weight merging and then refined with SFT. For inference speed, the authors also built Uno, a lossless diffusion adapter that does not require a separate Draft model.

Results

Limitations

The authors noted a performance gap on expert-level scientific reasoning such as HLE and CritPt. Comparisons are limited to some open and closed models from a similar period. The provided text is cut off partway, so full per-benchmark figures and the safety evaluation could not be confirmed.

Why it matters

By releasing the data pipeline and intermediate checkpoints, the work lets companies and researchers reproduce the training process or analyze the effects of scale and sparsity through controlled experiments. It also offers options from 0.9B to 375B for different deployment environments.

Key terms

MoVA(Mixture-of-Value Attention)
An architecture that places multiple experts in the attention Value computation, with a router selecting only some of them. Only the combined Value is cached, so the KV cache size does not grow.
Grouped RMSNorm
A method that splits the feature dimension into groups and normalizes each group separately. It stabilizes training by preventing normalization outputs from becoming too large in wide models.
Weight merging
A technique that combines the parameters of models trained separately for different domains into a single model. It keeps most of the performance gains each expert model achieved.
A23B notation
In an MoE model, it means 23B active parameters are actually used in computation when processing one token.

Original

Authors: K2 Team, Zhengzhong Liu, Haonan Li, Xuezhe Ma, Alexander Moreno, Subham Sekhar Sahoo, Rupesh Kumar Srivastava, Mikhail Yurochkin, et al. · Posted 2026-11-16 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.