Weili Xu (@_weilix) header
Weili Xu (@_weilix) profile picture

Building agentic and tiled stuff @UofIllinois | prev @togethercompute @ZJU_China

San Francisco, CA
Joined November 2023
Photo by Weili Xu (@_weilix)Photo by Weili Xu (@_weilix)Photo by Weili Xu (@_weilix)Photo by Weili Xu (@_weilix)Photo by Weili Xu (@_weilix)Photo by Weili Xu (@_weilix)Photo by Weili Xu (@_weilix)Photo by Weili Xu (@_weilix)
Weili Xu retweeted
Last Year We Built Megakernels. This Year We Deleted Them. One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target. Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput. Today, we’re sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization. Shout out to @dylan__lim @reyna_abhyankar @simran_s_arora and the team! Read more: together.ai/blog/road-to-tog…
38
71
20
745
77,461
We breathed life into LLM system. Introducing Evolving Infra⚙️: an inference framework that adapts diverse and dynamic agent workload purely online and retires fixed PD disaggregation. vs. NVIDIA Dynamo on up to 256 GB300 GPUs: ➡️ Up to 4.1× lower latency ➡️ Up to 14× lower P95 TTFT ➡️Up to 2.36× more revenue on real agent traces(Computer Use, Coding and math agents) and opensource models. 🌐 Website: inference-rsi.com 📄 Tech report: alphaxiv.org/en/abs/2611.123… 💻 Code: github.com/mit-han-lab/Evolv… Which agent(e.g. Pi/Openhands) do you want us to accelerate next? Tell us below 👇
21
60
15
271
56,308
Weili Xu retweeted
Replying to @fkysly
ai用多了,现实世界还会暴躁,习惯了命令、甚至颐指气使,而且对人的忍耐度会降低
2
1
46
8,155
Weili Xu retweeted
🤔 Is score centering (SC) all we need for stable async RL? People often stack truncated importance sampling (TIS) or KL penalties on top of SC. In a fun side project, we tested how much they help. We found that SC alone often matched on-policy. But when it fell short, adding more corrections doesn't necessarily help. 🧵
6
40
8
241
27,975
Weili Xu retweeted
FlashState is out ⚡️ Wild to see so much love for this! It started with a long car ride to SFO with @_weilix, chatting about ideas that grew out of Weili’s summer at @togethercompute, and here we are. So much fun co-leading this with Weili! Next up at @togethercompute : taking it from research to distributed production serving, and bringing it to more diverse models, agentic workloads, and newer hardware. Long road ahead, stay tuned! 🚀
Introducing FlashState ⚡: an inference system that flexibly manages the prefix cache of hybrid LLMs, designed for agentic workloads. On GLM 5.3 Flash, FlashState more than doubles vLLM's throughput and provider profit, while cutting latency by 3.7× and user cost by 2.5×.
2
7
18
797
Weili Xu retweeted
Is claude code usage nerfed for max 20x this week? Feeling like it depletes much faster...
4
1
12
1,188
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
94
142
45
967
175,287
Weili Xu retweeted
While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎 🌐 Project + live demos: introspective-diffusion.gith…
4
22
3
69
21,394
While QAT helps improve model quality at 2-4 bits, it has a structural optimization problem. Indeed, at every step, the high-precision latent weights receive the gradient updates, but the loss and gradients are computed on their low-bit reconstructions, i.e. the weights the model will use at inference. This mismatch results in suboptimal training dynamics and a higher loss floor. The main existing QAT methods (learning the scales alongside model parameters, soft-to-hard rounding, or better initialization) do not directly address this mismatch. With QUASAR, we introduce a simple and computationally cheap way to address the mismatch. QUASAR consistently achieves lower loss than all QAT baselines and adds no inference overhead. Also, it can be combined with most other QAT methods, since it only changes the weight reconstruction map in the fwd pass.
19
34
6
199
11,757
Weili Xu retweeted
Real-time video assistants have to decide when to respond and when to stay silent. Offline scores do not capture this. InteractionBench evaluates this decision for the complete system of model, memory, and response controller, with 1,060 interactions over 812 videos.
1
8
2
64
7,417
gpt 6 sol and gpt 6.1 sol are both sol bad
2
10
1,750
Weili Xu retweeted
We’re teaching a new course at Penn: CIS 6280 · World Models 🌍 cis.upenn.edu/~cis6280/ Every day, LLMs amaze me with one more thing they can solve. But I’ve always felt there’s more to learn than what humans have written down—from observing the world, interacting with it, and experiencing what happens next. To me, that’s what world models are about—and why I see them as one of AI’s next big bets. However, I struggled to find a systematic course on them—so we are building one! Topics will include: representation learning, generative models, simulation, model-based RL, video and 3D generation, world models for robotics, reasoning, and code-based world models. Hands-on work will include: - Building an environment - Training a world model - Learning a policy and a final research project with leaderboard. We’re already six lectures in. Slides, demos, and readings are public, and we’ll keep adding materials throughout the fall. Big thanks to our TA team— @TongMutianTMT @hagsaeng_bag @EnxinSong @KeelyAi04 —for helping bring this course to life!
58
367
27
2,390
578,316
Weili Xu retweeted
🎉Qwen3.8-27B 迎来最强 4-bit 量化版! 社区发布了基于量化感知训练(QAT)的:QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 🔥核心亮点 1. 极致压缩 全网络 496 个线性层全面采用 NVFP4(W4A4)量化,模型体积从 55.6GB → 19.7GB,大幅降低显存占用 2. 近乎无损性能 通过蒸馏学习,保留原版几乎 100% 的推理与数学能力,表现远超传统 PTQ 量化方案 3. 即插即用 原生支持 vLLM / SGLang,无需额外转换,直接部署运行 4-bit 不再只是能跑就行,而是开始追求接近原版的性能了!
🎉Qwen 3.8 27B 变身 20GB!Blackwell 架构完美适配版! 社区又又又推出了 Huihui-Qwen3.8-27B-abliterated-NVFP4 最新量化版本! 🔥核心亮点: 1. 体积暴减: 顺应 8 月新发布的 Qwen 3.8 27B 基座,将 55.6 GB 降至 20.0 GB,双 16GB 显卡即可留足 KV 缓存轻松跑起来,无压力 2. 修复长思考 Bug: 完美解决上一版在 Qwen 3.8 标志性的 xhigh 深度思考模式下,因思维奔逸陷入万字复读(thinking-runaway)的致命缺陷 3. MTP 投机采样: 完好保留 bf16 格式的 MTP 团队草稿头,在 RTX PRO 2000 Blackwell 上单流 Prefill 飙到 3,250 tok/s 划重点 部署提示: 基于 vanilla vLLM v0.22.0,直接 vllm serve 自动识别 compressed-tensors,无需额外量化 flag 无论你是想本地调教还是用作生产力 这个版本都是性价比之王 Qwen 牛牪犇! 传送门👇🏻
41
95
6
735
192,192
Recently I was reading Jianlin Su’s blog and one statement caught me off guard: MLA + MTP may actually be a bad combination. That sounded counterintuitive, so I tried to understand why from the perspective of arithmetic intensity. Following the path from MHA → GQA → MQA → MLA, the result turned out to be surprisingly elegant. Check Full Blog Here!changyi.fun/posts/attention-… A short thread on both:
🤖 Made with AI
14
24
2
144
14,023
Cognition used Dynamo to accelerate the rollout phase of agentic RL. Our work ThunderAgent has also been merged into Dynamo and SkyRL. And achieve lossless speed up up to 4x. However, current schedulers and KV-cache managers are not yet designed for hour-scale agentic rollouts, leaving significant room for improvement in trajectory-aware scheduling, KV lifecycle management, and distributed memory pooling. paper: arxiv.org/pdf/2602.13692 code: github.com/ThunderAgent-org/… web: thunderagent.ai/
Introducing SWE-1.7, the most capable model we’ve trained yet. It scores within a few points of the strongest frontier models at a fraction of the cost, and is now available at 1000 tok/s. RL is not hitting its limit: after refining our recipe, we keep seeing gains as we scale
5
8
78
9,710
Our research team has 9 papers at ICML next week! Spanning the full stack from frontier agents to GPU kernels, we're excited to share what our researchers and collaborators have been working on. If you're in Seoul for ICML come meet our team or catch a talk! 🧵
5
12
75
4,758
2/ ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System Paper: arxiv.org/abs/2602.13692
3
1
4
808
Multi-GPU kernels are the real test for coding models. Today at @aiDotEngineer, @simran_s_arora shared ParallelKernelBench, an open-source benchmark for evaluating whether LLMs can write fast CUDA kernels for real communication-heavy workloads. Proud to see this work from the Together AI Frontier Performance team.
8
30
247
17,180