头雁|Nov 22, 2026 12:07
The LLM training technical report released by K2 Horizon (primarily a Chinese team) is characterized by its extreme transparency and systematic disclosure of training details, far exceeding most open-source LLM reports. It adheres to the LLM360 standard, fully disclosing "how it was done," rather than just providing the final weights.
- Extreme Openness and Reproducibility: Releases the entire training history (380 intermediate checkpoints, covering all stages), 21.6TB of pre-training data, 58 training run logs, and 17 code repositories (including a production-grade xLLM framework, RL360 reinforcement learning stack, Eval360 evaluation, etc.). It not only provides data but also details the complete pipeline and decision-making process for generating approximately 10TB of synthetic tokens, a part that most open-source reports avoid.
- Architectural and Inference Innovations: Proposes Mixture-of-Value Attention (MoVA, applying MoE to attention value projection) and softplus attention gating for enhanced inference.
- One particularly noteworthy technique is the use of a method from the paper ISO: An RLVR-Native Optimization Stack in the post-training stage. This method merges multiple RL specialists trained on the same base model at frame coordinates with a fixed spectrum. It eliminates the need for post-merging, rollout, gradient manipulation, or distillation, outperforming common data-nothing merging methods like Task Arithmetic, TIES, and RAM. https://iso-rlvr.(github.io)/
https://arxiv.org/abs/2611.21003
Share To
HotFlash
APP
X
Telegram
CopyLink