
LLM Post-training Lead @IFM_AI. Previously (co)developed Highway Networks, Upside-Down RL, Bayesian Flow Networks, EvoTorch.
Santa Cruz, CA
Joined September 2014
- Tweets1.2K
- Following795
- Followers2.7K
- Likes2.5K
Pinned Tweet
Do You Like Open Source AI?
huggingface.co/collections/I…
We just released a fleet of 6 great LLMs:
0.9B, 4B, 7B, 32B dense, 36 MoVA, 375B MoE
GGUFs and FP8 quantized variants
New "Uno" adapters for fast diffusion decoding!
Loads of training data, logs, intermediate ckpts for pre-training, midtraining, post-training.
Introducing K2 Horizon: a connected fleet of six foundation models ranging from 0.9 billion to 375 billion parameters.
- Frontier performance: Across coding and agentic tasks, K2 Horizon delivers top-tier performance in every size class—with the 0.9B, 3.7B and 7B models setting new state of the art at their respective scales.
- Radical openness: K2 Horizon represents the largest fully open-source model launch in AI history. The fully open code, training data and recipes are a significant step forward in transparency.
Launch page: ifm.ai/k2/
Tech blog: ifm.ai/blog/k2
Hugging Face: huggingface.co/collections/I…
Maximus Openus LLM training
The most detailed open LLM data release yet: K2 Horizon's pre-training data and post-training data-generation pipelines are now open!
13 trillion pre-training tokens across 5 datasets and 24 subsets. Over 10 trillion tokens of synthetic data spanning mathematics, reasoning, coding, planning, and problem-solving.
Beyond the scale, four highlights for model builders (details in the threads):
- Scaling synthetic data without losing diversity. Our generation methods produce diverse training examples at scale, nearly matching natural web text in measured diversity.
- More verification isn't always better. Strict verification can throw away useful training signal.
- Small-scale wins can mislead. Our chosen data mix actually lost in smaller experiments.
- Beyond teacher imitation. Our post-training data generation methods produce training trajectories the teacher couldn't generate on its own.
Open pre-training datasets, post-training data-generation pipelines, and processing tools for researchers to inspect, reuse, and build on.
Pre-training data: huggingface.co/collections/I…
Post-training data-generation code & tools: github.com/ifm-ai/DATA_NAVIG…
Rupesh Srivastava retweeted
🎊 Excited to give a tutorial on diffusion LLMs at @NeurIPSConf 2026 with @StefanoErmon @JCJesseLai @mittu1204
📆 Mon, Dec 7, 9 AM AEDT
📍 TikTok Ent Centre (Sydney, Australia)
Topics covered:
> A beginner friendly introduction to diffusion-LLMs
> Designing frontier diffusion-LLMs
> Using Continuous diffusion for LLMs
Rupesh Srivastava retweeted
8,192 concurrent sandboxes and ~1.3 million rollouts. Really impressive engineering from @IFM_AI!
One thing I find particularly interesting about long-horizon agentic RL is that infrastructure failures can directly affect training correctness. If a sandbox crashes halfway through a trajectory and the system treats it as a failed task, we're effectively penalizing the model for something it never did wrong. At this scale, distinguishing environment failures from actual model failures becomes part of the learning algorithm itself.
RL360 also demonstrates why modular RL infrastructure matters. With Miles handling rollouts and optimization, Harbor managing agent environments, SGLang serving inference, and Megatron-LM running training, each component can evolve and scale independently.
I'm especially happy to see Miles and SGLang being used together in such a large-scale, real-world RL system. The lessons about asynchronous execution, distributed trajectory management, and sandbox reliability are incredibly valuable for the entire open-source community.
Huge congratulations to the IFM team!
github.com/ifm-ai/RL360
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts.
Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer.
Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes.
RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread.
Code: github.com/ifm-ai/RL360
Run: wandb.ai/mbzuai-llm/rl360-pu…
Rupesh Srivastava retweeted
Agentic RL is a whole new beast. Things broke. A lot.
Pods crashed. Registries choked. Runs stalled.
I spent the last few months in the trenches isolating failures and fixing bottlenecks. RL360 is what came out of it: millions of rollouts, scaled across multiple experts.
Proud of this one. Here's how 👇
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts.
Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer.
Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes.
RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread.
Code: github.com/ifm-ai/RL360
Run: wandb.ai/mbzuai-llm/rl360-pu…
We learned a TON in the last 6 months about Agentic RL infrastructure at scale, and now we're happy to share it with everyone.
RL360 was used for the training of K2 Horizon series (artificialanalysis.ai/models…). It builds on top of miles from @radixark, adding a lot of optimizations for efficient agentic RL at scale.
Our amazing team at @IFM_AI built this with a lot of hard work. Thanks to @varad0309 @willis_guo @wang1999_zt @mingshanhee @DavidRBellamy @omkarenator @_matthewyang @qiancheng1231 @bhandari_jalaj and many others!
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts.
Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer.
Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes.
RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread.
Code: github.com/ifm-ai/RL360
Run: wandb.ai/mbzuai-llm/rl360-pu…
One of these models is not like the others!
@IFM_AI is not a major tech company or a highly-funded startup. K2 Horizon was our first time building an open agentic model from scratch, with a very small team. It comes with open pretraining data, data pipelines, RL codebase, regularly spaced intermediate checkpoints, and a very detailed tech report coming soon!
We never evaluated on FrontierCode so it was a nice surprise to see these results :)
We're launching our leaderboard for FrontierCode 1.1, a private evaluation from Cognition that tests models on completing coding tasks in a way that would actually get merged by maintainers of real-world repositories
@cognition announced FrontierCode in June 2026 and updated it to version 1.1 in July. Cognition built the dataset with open source maintainers who spent more than 40 hours per task. Each task requires the agent to make a change to a real open source repository, then checks both unit tests and qualitative tests for what maintainers would require to merge the change.
We run FrontierCode 1.1 Main, the primary 100-task set of the benchmark's 150 tasks, using our Stirrup open source reference harness for apples-to-apples comparisons across models. All tasks are private to avoid contamination. Each attempt is graded with both blocking and non-blocking criteria. An attempt that fails any blocking check scores zero, and one that passes them all scores between 70% and 100%.
Unlike other coding evaluations, FrontierCode has a more holistic view of success based on real usage. Models need to pass unit tests, but are also assessed on behaviors like making excessive changes and large diffs, modifying files outside the relevant scope, or adding unhelpful tests. We think this approach is valuable in the current context: models are now very capable at coding tasks, but their changes can still require substantial cleanup before a maintainer would merge them.
Key takeaways:
➤ Leading models still fail more than 40% of the time: Claude Opus 5.5 (max with fallback) leads at a score of 52% after passing blockers in 58% of its attempts. Nine of the 100 tasks were not passed in any attempt by any of the models we evaluated for launch
➤ The top of the leaderboard is tightly packed: nine models from six labs score between 43% and 52%. MiMo-V2.6-Pro is the leading open weights model at 45%, followed by DeepSeek V4.1 Flash (max) and Kimi K3 (max) at 44%
➤ GPT-6.1 Sol (max) offers the strongest performance for its cost: At $0.81 per task it scores 49%, matching Claude Fable 5.1 (max with fallback). This means it achieves similar performance to leading Anthropic models, at less than a tenth of the cost per task
Thank you to @silasalberti, @ybenpan, @cl571128, and the rest of the team at Cognition for building FrontierCode and collaborating with us to bring it to Artificial Analysis - we’re working to add FrontierCode to our coding agent leaderboard very soon.
Rupesh Srivastava retweeted
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Shaping fixed points in looped language models lets a 1.6B parameter model with a 3x smaller key-value cache match the downstream accuracy of a full-cache fixed-depth baseline.
arxivb.org/abs/2610.06833
Rupesh Srivastava retweeted
Most decisions don't need words. 🤫⚡
Meet K2-Type-0.9B 🧠: an open decision model. One state + many typed questions in, calibrated probabilities out, in a single forward pass. No text generated.
Apache-2.0 weights 👇
huggingface.co/IFM/K2-Type-0…
#OpenSource #LLM #AI
Rupesh Srivastava retweeted
Grouped-query attention (GQA) reduces KV-cache memory by sharing keys and values across attention heads. More KV groups give more independent representations, at the cost of larger cache.
MoVA adds capacity within the existing groups instead. Sparse value experts introduce additional parameters and nonlinearity into the value computation, while the value vectors keep their original dimensions. Each token selects 4 of 64 value experts, activating just 6.25% of the value-expert pool. The value matrix gains modeling capacity at no additional KV-cache cost.
Rupesh Srivastava retweeted
The MoVA model is here: huggingface.co/IFM/K2-Horizo…
We are going to release the code and technical report soon.🔥
K2-Horizon-36B-A4B scores 25 on the Artificial Analysis Intelligence Index, matching models with over 20× the total parameters while using 4B active parameters per token.
These capabilities come from our new architecture MoVA (Mixture-of-Value Attention), which incorporates MoE-based sparsity into the compute of value vectors in multi-head attention.
It opens a second axis for scaling sparsity in an LLM, beyond MoE in the FFN module. Importantly, MoVA enjoys the following advantages:
• Simple and compatible with efficient attention algorithms, such as flash attention, GQA, and sparse attention
• No additional KV cache cost comparing to standard GQA
K2-Horizon-36B-A4B available at: huggingface.co/IFM/K2-Horizo…
MoVA enables our 36B total 4B active model to get close to our 32B dense!
K2-Horizon-36B-A4B scores 25 on the Artificial Analysis Intelligence Index, matching models with over 20× the total parameters while using 4B active parameters per token.
These capabilities come from our new architecture MoVA (Mixture-of-Value Attention), which incorporates MoE-based sparsity into the compute of value vectors in multi-head attention.
It opens a second axis for scaling sparsity in an LLM, beyond MoE in the FFN module. Importantly, MoVA enjoys the following advantages:
• Simple and compatible with efficient attention algorithms, such as flash attention, GQA, and sparse attention
• No additional KV cache cost comparing to standard GQA
K2-Horizon-36B-A4B available at: huggingface.co/IFM/K2-Horizo…
This model has been exceeding our expectations since pre-training. Just one of those lucky ones 🙃
K2-Horizon-7B scores 21 on Artificial Analysis Intelligence Index, punching above its weights.
It scores 50% above the next best model under 10B, beating Qwen3.6-35B-A3B and closely matching Qwen3.6-27B.
• Try it: huggingface.co/IFM/K2-Horizo…
• 8-bit quantization: huggingface.co/IFM/K2-Horizo…
• Uno, the speed-up diffusion-augmented version: huggingface.co/IFM/K2-Horizo…
Rupesh Srivastava retweeted
Rupesh Srivastava retweeted
🎊 Uno is on Hugging Face, go try it out: huggingface.co/IFM/K2-Horizo…
Diffusion LLMs have two limitations relative to AR models:
(1) Lower quality, and
(2) Slower inference at large batch sizes.
We address this "Uno"
> Retains the AR architecture of LLMs
> Each layer has two sets of weights: AR weights and Diffusion weights
> Diffusion weights enable parallel sampling from the AR distribution losslessly
Results:
💥Faster than all speculative decoding methods: DFlash and EAGLE-3
🔥 Beats ALL diffusion LLMs: Mercury 2, Diffusion Gemma, Llada
Links to the paper, models, code below
Rupesh Srivastava retweeted
Great to see Neural Data Router (ICLR 2022) also covered here.
By @robert_csordas ahead of time, now popular: improving reasoning & generalization with shared-layer / looped / universal transformers.
Arguing _why_ a "recurrent loop" helps + intuitive algorithmic illustration.
Reproducing all of Schmidhuber’s papers (1990-2025) using an AI coding assistant.
Cool project by @yaroslavvb! It even reproduced the “World Models” paper by me and @SchmidhuberAI with a toy env, with a full VAE + RNN world model implementation.
Project: github.com/cybertronai/schmi…
Rupesh Srivastava retweeted
K2 Horizon 375B A23B, a new open weights model from UAE's MBZUAI, scores 47 on the Artificial Analysis Intelligence Index, with relatively strong agentic performance and a 30 point jump over its predecessor
K2 Horizon 375B A23B is an open weights Mixture-of-Experts model with 375B total and 23B active parameters from @IFM_MBZUAI, MBZUAI's Institute of Foundation Models. It scores 47 on the Intelligence Index, alongside models such as MiniMax-M3 (45, also a MoE with 23B active parameters), and a large upgrade from its predecessor K2 Think V2 (17, 70B dense model). It leads nearby open weights models on agentic evals and has a low hallucination rate, but trails on knowledge and the hardest reasoning evals. K2 Think V2 ranks among the most open models on our Openness Index; MBZUAI is updating the supporting documentation and code for K2 Horizon and we expect to add it to the Openness Index soon.
Key takeaways:
➤ Strong on agentic tasks, weaker on knowledge and deep reasoning. MiniMax-M3, a recent model that is close to it on the Intelligence Index, makes the cleanest comparison: K2 Horizon 375B A23B leads on GDPval-AA, our real-world knowledge work benchmark (Elo 1430 vs 1380), and on τ³-Banking (34.2% vs 15.3%), but trails on GPQA Diamond (87.3% vs 92.9%) and Humanity's Last Exam (32.0% vs 39.0%)
➤ Low hallucination rate, driven by abstention rather than knowledge. K2 Horizon 375B A23B attempts only 40% of AA-Omniscience questions, declining the remaining 60% rather than guessing. The result is a 26% hallucination rate, among the lower rates we have measured, while accuracy is 18%, essentially unchanged from K2 Think V2
➤ A new architecture over its predecessor. K2 Horizon 375B A23B is a 375B parameter Mixture-of-Experts model with 23B active, succeeding the 70B dense K2 Think V2, and extends context from 262K to 512K tokens. Its 23B active parameters match MiniMax-M3 (428B total, 23B active)
Key model details:
➤ Architecture: Mixture-of-Experts, 375B total parameters, 23B active
➤ Context window: 512K tokens
➤ Multimodality: Text input and output only
➤ Pricing and availability: Yet to be announced
➤ Licensing: Open weights (license details to be announced)



































