Connor Li (@connorliml) header
Connor Li (@connorliml) profile picture

Research Engineer @togethercompute | Prev @CMU | | First author of FlashStates

San Francisco
Joined November 2026
FlashState is out ⚡️ Wild to see so much love for this! It started with a long car ride to SFO with @_weilix, chatting about ideas that grew out of Weili’s summer at @togethercompute, and here we are. So much fun co-leading this with Weili! Next up at @togethercompute : taking it from research to distributed production serving, and bringing it to more diverse models, agentic workloads, and newer hardware. Long road ahead, stay tuned! 🚀
Introducing FlashState ⚡: an inference system that flexibly manages the prefix cache of hybrid LLMs, designed for agentic workloads. On GLM 5.3 Flash, FlashState more than doubles vLLM's throughput and provider profit, while cutting latency by 3.7× and user cost by 2.5×.
2
7
18
797
Meet @Mlsys_HaoKang and read this paper few days ago; the idea of ​​"unblocking infra's flexibility" sounds absolutely wonderful, which also is a foundational work for Agent Tuning Infra. Amazing!
We breathed life into LLM system. Introducing Evolving Infra⚙️: an inference framework that adapts diverse and dynamic agent workload purely online and retires fixed PD disaggregation. vs. NVIDIA Dynamo on up to 256 GB300 GPUs: ➡️ Up to 4.1× lower latency ➡️ Up to 14× lower P95 TTFT ➡️Up to 2.36× more revenue on real agent traces(Computer Use, Coding and math agents) and opensource models. 🌐 Website: inference-rsi.com 📄 Tech report: alphaxiv.org/en/abs/2611.123… 💻 Code: github.com/mit-han-lab/Evolv… Which agent(e.g. Pi/Openhands) do you want us to accelerate next? Tell us below 👇
3
171
Hybrid Attention models are becoming the default for frontier serving, but prefix caching was built for pure attention. Under agentic workloads it breaks: a KV hit is useless without the matching recurrent state, so every turn re-prefills. FlashState checkpoints that state exactly where the next turn resumes, and evicts by how much prefill each state saves. No tuning. On 48 Claude Code-style agents (2×B200 per engine): → up to 2.9× output TPM vs vLLM → 97% fewer uncached tokens → 87% lower p90 TTFT Tested on GLM-5.3-Flash (@Zai_org), Nemotron 3 Super (@NVIDIAAI) and Qwen3.8-Next (@Alibaba_Qwen), plus a Kimi-K3.1-Pro (@Kimi_Moonshot) build on GB300. Benchmarked on the open AgentX traces and harness from @SemiAnalysis_. Huge thanks for making agentic serving measurable. Would love to see hybrid-model caching in InferenceX! #LLM #inference #AgentX #infernceX
Introducing FlashState ⚡: an inference system that flexibly manages the prefix cache of hybrid LLMs, designed for agentic workloads. On GLM 5.3 Flash, FlashState more than doubles vLLM's throughput and provider profit, while cutting latency by 3.7× and user cost by 2.5×.
2
3
15
63,587
Hi, There!
1
3
137