
Research Engineer @togethercompute | Prev @CMU | | First author of FlashStates
San Francisco
Joined November 2026
- Tweets19
- Following71
- Followers59
- Likes48
Pinned Tweet
FlashState is out ⚡️
Wild to see so much love for this! It started with a long car ride to SFO with @_weilix, chatting about ideas that grew out of Weili’s summer at @togethercompute, and here we are.
So much fun co-leading this with Weili!
Next up at @togethercompute : taking it from research to distributed production serving, and bringing it to more diverse models, agentic workloads, and newer hardware.
Long road ahead, stay tuned! 🚀
Meet @Mlsys_HaoKang and read this paper few days ago; the idea of "unblocking infra's flexibility" sounds absolutely wonderful, which also is a foundational work for Agent Tuning Infra. Amazing!
We breathed life into LLM system.
Introducing Evolving Infra⚙️: an inference framework that adapts diverse and dynamic agent workload purely online and retires fixed PD disaggregation.
vs. NVIDIA Dynamo on up to 256 GB300 GPUs:
➡️ Up to 4.1× lower latency
➡️ Up to 14× lower P95 TTFT
➡️Up to 2.36× more revenue
on real agent traces(Computer Use, Coding and math agents) and opensource models.
🌐 Website: inference-rsi.com
📄 Tech report: alphaxiv.org/en/abs/2611.123…
💻 Code: github.com/mit-han-lab/Evolv…
Which agent(e.g. Pi/Openhands) do you want us to accelerate next? Tell us below 👇
3
171
Hybrid Attention models are becoming the default for frontier serving, but prefix caching was built for pure attention. Under agentic workloads it breaks: a KV hit is useless without the matching recurrent state, so every turn re-prefills.
FlashState checkpoints that state exactly where the next turn resumes, and evicts by how much prefill each state saves. No tuning.
On 48 Claude Code-style agents (2×B200 per engine):
→ up to 2.9× output TPM vs vLLM
→ 97% fewer uncached tokens
→ 87% lower p90 TTFT
Tested on GLM-5.3-Flash (@Zai_org), Nemotron 3 Super (@NVIDIAAI) and Qwen3.8-Next (@Alibaba_Qwen), plus a Kimi-K3.1-Pro (@Kimi_Moonshot) build on GB300.
Benchmarked on the open AgentX traces and harness from @SemiAnalysis_. Huge thanks for making agentic serving measurable. Would love to see hybrid-model caching in InferenceX!
#LLM #inference #AgentX #infernceX


