Vikranth Srivatsa retweeted
So much went into building Together Compiler! Today, we're sharing our journey with the broader community. If you're curious about the inner workings of Torch Compile, megakernels, and earlier iterations of our agentic optimizer, look no further :)
Couldn't have done this without @dylan__lim and @simran_s_arora!
Last Year We Built Megakernels. This Year We Deleted Them.
One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target.
Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
Vikranth Srivatsa retweeted
I left my PhD 16 months ago to figure out how to scale AI inference optimization. Today, I’m proud to announce the release of Together Compiler! It's the first continual inference optimization system that adapts to new-age architectures and diverse user behavior. The first system designed for the age of “polymorphic” inference.
Together Compiler generates optimizations specialized to a workload and posts best-in-class TPS numbers on highly-optimized open source benchmarks: 1100+ TPS on MiniMax M3.1 Flash and 1200+ TPS for DeepSeek v4.1 Flash off InferenceX. 24 hours and just 2 nodes per model. That’s it!
Paper: alphaxiv.org/abs/2611.togeth…
It was super exciting to build this from scratch :) more thoughts in thread below!! 1/
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
Vikranth Srivatsa retweeted
In today's LLM serving, a single system handles requests with different demands. We built a multi-tier-SLO LLM serving system that treats Tensor Parallelism as a runtime knob instead of a fixed setting. Result: 5.3x better SLO-compliant goodput than SoTA.
Vikranth Srivatsa retweeted
🔥CAD: Efficient Long-context Language Model Training by Core Attention Disaggregation
Repo: github.com/hao-ai-lab/distca
Blog: hao-ai-lab.github.io/blogs/d…
Training a long-context LLM model can suffer from severe workload imbalance caused by core-attention - the softmax(QK^T)V part.
Core-attention disaggregation (CAD) fundamentally eliminates workload imbalance by disaggregating core-attention from the rest of the model.
Vikranth Srivatsa retweeted
🔥 New Blog: “Disaggregated Inference: 18 Months Later”
18 months in LLM inference feels like a new Moore’s Law cycle – but this time not just 2x per year:
💸 Serving cost ↓10–100x
🚀 Throughput ↑10x
⚡ Latency ↓5x
A big reason? Disaggregated Inference.
From DistServe, our early research system on prefill-decode disaggregation, to today’s production frameworks, disaggregation has become the backbone of modern LLM serving.
So what is disaggregated inference?
Why does the LLM inference community love it?
And how far have we come?
As the inventors of this technique, we take a look back – 18 months later - at how the idea reshaped the landscape and what comes next.
🔗 Read the full story: hao-ai-lab.github.io/blogs/d…
Vikranth Srivatsa retweeted
WukLab's new study reveals CPU scheduling overhead can dominate LLM inference time—up to 50% in systems like vLLM! Scheduling overhead can no longer be ignored as model forwarding speeds increase and more scheduling tasks get added.#LLM #vLLM #SGLang
Read tinyurl.com/yk4jeaz8
Vikranth Srivatsa retweeted
Today, LLMs are constantly being augmented with tools, agents, models, RAG, etc. We built InferCept [ICML'24], the first serving framework designed for augmented LLMs. InferCept sustains a 1.6x-2x higher serving load than SOTA LLM serving systems. #AugLLM
mlsys.wuklab.io/posts/inferc…
Vikranth Srivatsa retweeted
LLM prompts are getting longer and increasingly shared with agents, tools, documents, etc. We introduce Preble, the first distributed LLM serving system targeting long and shared prompts. Preble reduces latency by 1.5-14.5x over SOTA serving systems. #LLM
mlsys.wuklab.io/posts/preble…












