Dylan Lim (@dylan__lim) header
Dylan Lim (@dylan__lim) profile picture
Stanford, CA
Joined April 2024
Photo by Dylan Lim (@dylan__lim)Photo by Dylan Lim (@dylan__lim)Photo by Dylan Lim (@dylan__lim)Photo by Dylan Lim (@dylan__lim)Photo by Dylan Lim (@dylan__lim)Photo by Dylan Lim (@dylan__lim)Photo by Dylan Lim (@dylan__lim)Photo by Dylan Lim (@dylan__lim)Photo by Dylan Lim (@dylan__lim)Photo by Dylan Lim (@dylan__lim)
Last Year We Built Megakernels. This Year We Deleted Them. One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target. Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput. Today, we’re sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization. Shout out to @dylan__lim @reyna_abhyankar @simran_s_arora and the team! Read more: together.ai/blog/road-to-tog…
33
62
17
592
46,853
Dylan Lim retweeted
Checkout the video in @dylan__lim's post to see the hidden bubbles: Look Ma, Some Bubbles Building on a long and wonderful tradition in Hazy lab of always investigating both sides (tb to S4 no attention vs FlashAttention)
Last Year We Built Megakernels. This Year We Deleted Them. One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target. Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
2
10
1
43
2,238
Dylan Lim retweeted
Replying to @dylan__lim
Thank you for showing us the way
1
2
8
537
Dylan Lim retweeted
Last Year We Helped Build Megakernels. 🔨 This Year We Deleted Them. ❌ Check out our post and new blogs for a deepdive into how we built the Together Compiler and the lessons learned along the way! Some key takeaways: 1. When tuned appropriately, simply using Torch Inductor performs as well as hand-written Megakernels on canonical Megakernel workloads. 2. Full-model Megakernels are the wrong target to blindly hill-climb on. Fusion can be harmful. 3. The optimal abstraction for inference optimization is neither individual kernels nor one giant megakernel. And a bunch of interesting lessons on how agents interacted with various scaffoldings, compiler intermediate representations, and "expert guidance" we baked into the harness.
Last Year We Built Megakernels. This Year We Deleted Them. One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target. Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
1
21
97
5,026
Dylan Lim retweeted
Last Year We Built Megakernels. This Year We Deleted Them. One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target. Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput. Today, we’re sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization. Shout out to @dylan__lim @reyna_abhyankar @simran_s_arora and the team! Read more: together.ai/blog/road-to-tog…
33
62
17
592
46,853
So much went into building Together Compiler! Today, we're sharing our journey with the broader community. If you're curious about the inner workings of Torch Compile, megakernels, and earlier iterations of our agentic optimizer, look no further :) Couldn't have done this without @dylan__lim and @simran_s_arora!
Last Year We Built Megakernels. This Year We Deleted Them. One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target. Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
2
5
16
880
Dylan Lim retweeted
Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput. Today, we’re sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization. Shout out to @dylan__lim @reyna_abhyankar @simran_s_arora and the team! Read more: together.ai/blog/road-to-tog…
9
22
2
130
37,864
Through these findings, we realized that an optimal compiler is not always a megakernel compiler. Many people are working on this problem and megakernels are hotter than ever before. Given this, it’s important to emphasize that rather than fusing everything into one persistent kernel, the more important task is to identify the right group of operations to couple and individually tune. While this broadens the search space, agentic systems are increasingly well-suited to navigate it. We believe that agents will become a core component of the modern compiler stack, and Together Compiler is our effort to demonstrate this. With the right system in place, simple primitives can achieve SOTA results on real world inference workloads.
1
3
27
1,893
For the past two years, members of our team have been investigating what a next-generation system for inference optimization should look like. In the following blog, we revisit the assumptions behind today's compilers, share the experiments that challenged our thinking, and explain the lessons that shaped our compiler. This work was made possible through my collaborators, @reyna_abhyankar and @simran_s_arora and things are still incredibly early! There’s many more findings to share and interesting work to be done. Stay tuned! Blog: together.ai/blog/road-to-tog…
1
9
1
48
4,431
Dylan Lim retweeted
"Together Compiler: Continual Compilation for AI Inference" AI agents can optimize LLM inference, but testing every change requires restarting the entire inference engine, making experimentation extremely slow. This paper by Together AI introduces Together Compiler, which lets coding agents continuously modify and test inference code without restarting the engine, using real workloads to measure performance. This enables 252x more agent experiments and achieves up to 2.6x higher inference throughput than top open-source engines across frontier models. alphaxiv.org/abs/2611.togeth…
13
28
4
152
10,264
Replying to @simran_s_arora
amazing work!
1
2
9
1,399
Dylan Lim retweeted
Congratulations to @reyna_abhyankar and @dylan__lim for their >1yr effort on Together Compiler! Their work was so incredible that I found it generalized well beyond LLMs to SciML models 🔬. Here it is outperforming Claude on nearly every model I tried 1/n
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
1
4
2
19
3,068
Dylan Lim retweeted
Distilling most of our expertise in kernel and inference engine optimization into an agent loop that powers our production. A 2-year project that we're finally able to share
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
21
42
1
481
21,546
“AI is now improving the systems that run AI.” Thanks to @Forbes for covering Together Compiler and the shift to Polymorphic Inference. New open models can now get fully optimized for production performance in hours, not weeks. togetherai.link/OeAIz0u
5
12
71
5,014
Dylan Lim retweeted
Great work that pushes our understanding of how agents and experts can collaborate on optimizing system performance Beyond the strong results, the team also found some surprising cases where our human inductive biases end up limiting what agents can achieve – at East Palo Alto, CA
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
19
13
1
229
40,108
Together Compiler discovers the most efficient inference runtimes with continuous compilation for any combination of workload, model and silicon. Excited to share the details about the research and engineering that went into it. Together Compiler (or TKCC) is a production service that is deployed and operational across portions of our inference fleet, and optimizing our inference stack continuously. We think of this as a shift in inference technology from Static Inference (inference engines, hand-tuned configs, megakernels etc), to Polymorphic Inference, which are inference systems that continuously and autonomously adapt to workload, hardware and models, and are essential for achieving vast scale alongside the best possible utilization and economics. A thread from @simran_s_arora, who along with the research team at @togethercompute spearheaded the work on Together Compiler.
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
2
3
1
40
2,333
Dylan Lim retweeted
Super excited to (officially) share Together Compiler with the world! This is our opinionated take at @togethercompute about how to scale inference in an increasingly polymorphic world (many models, archs, traffic, hardware). Deployed across our inference fleet today 🚀 1/
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
7
13
1
79
5,623
Had a great time working on this! The future of systems is in agents that can continuously optimize and target the workloads that they serve.
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
11
5
41
1,556