@UWSyFi

The Systems for Future Intelligence lab at @uwcse.

 Seattle, Washington
syfi.cs.washington.edu/
Joined February 2026
  • Tweets74
  • Following16
  • Followers133
18Media
it was really fun to build memory for agents so they operate apps more effectively over time! we are actively working on scaling up the work. bigger deployments, more types of systems to operate, and more ways to use agents to improve system reliability. stay tuned!
Ratul Mahajan
@ratulmOct 23
The full paper on our self-defining operator is now out. We show how an agentic operator that remembers past incidents and synthesizes its own playbooks cuts cost and MTTR by over half. arxiv.org/abs/2610.14547

A Self-Defining Operator for Distributed Applications

To operate distributed applications with high availability, operators must rapidly diagnose and mitigate any incident. LLM-based agents can help, but unlike experienced human operators they do not...

arxiv.org
3
5
139
UW SyFi retweeted
Yi Pan
@conlesspanOct 21
Excited to share our recent work on leveraging TorchDynamo in AI training and inference systems at @UWSyFi with @thepadawang at #PyTorchConf today! ⚡️ We'll show how to use TorchDynamo beyond torch.compile, as a layer to accelerate, compare, and debug ML systems without rewriting the model. 🗓 5:30–5:55 PM PT 🏛 LL21A, San Jose Convention Center
7
2
22
945
Excited to announce that SyFI was just selected for a multi-million-dollar award from the @NSF to participate in the AI Efficiency Challenge over the next two years! We’ll be working with our industry partners on realizing the SyFI mission: hyperspecializing infrastructure for AI, by AI. The goal is to use AI to hyperspecialize AI system software to each model, workload, and hardware platform to unlock the next level of efficiency gains. Special thanks to our deployment partners @NVIDIA, @AMD, @Amazon, @mirendil for helping make this happen!
1
3
11
437
More information here: stride-ventures.com/start2-g…
91
1/ We pointed VibeSys, our multi-agent system for designing systems, at Qwen3.5-397B-A17B on four AMD MI300As. No human wrote engine code. In 105 hours, it went from a 12.8 tok/s PyTorch baseline to 2,242 tok/s goodput, 2.33× SGLang on the same hardware!
2
10
4
29
3,573
more replies
6/ Other optimizations: a fused MXFP4 MoE kernel, a fused all-reduce + RMSNorm, and MTP speculative decoding.
1
1
361
7/ Full write-up: syfi.cs.washington.edu/blog/… VibeSys: github.com/uw-syfi/vibesys Agent-built engine: github.com/vibesys-playgroun… Tuned SGLang: github.com/vibesys-playgroun… This project was led by @sudopowr, @KeisukeKamahori, Simon Peter, and @bariskasikci
1
5
404
Other optimizations: a fused MXFP4 MoE kernel, a fused all-reduce + RMSNorm, and MTP speculative decoding. (6/n)
1
23
Full write-up: syfi.cs.washington.edu/blog/… VibeSys: github.com/uw-syfi/vibesys Agent-built engine: github.com/vibesys-playgroun… Tuned SGLang: github.com/vibesys-playgroun… (7/n)
17
We first used VibeSys to port and tune SGLang (961 tok/s), and compare against that. Stock SGLang does not run this model on MI300A: no MXFP4 MoE kernel for gfx942, memory sizing that assumes discrete GPU memory, and a 60-minute weight load. (4/n)
1
35
The biggest single gain while the agents built their own engine was snapshot eviction: +27% goodput over the previous build. Qwen3.5 is 45 Gated DeltaNet layers + 15 full-attention layers. Each DeltaNet snapshot is ~47 MiB, and MI300A's HBM is shared with everything else. The prefix cache kept stale interior snapshots, evicting active conversations, whose next turns recomputed the full history. Evicting stale snapshots first cut prefill tokens by ~40%. (5/n)
1
28
The objective was peak goodput: throughput of responses that meet the latency SLO (p95 TPOT ≤ 250 ms, p95 TTFT ≤ 10 s) on a multi-turn chat workload that ramps from 1 to 192 concurrent sessions. (2/n)
1
43
The agents received the weights, the HF reference implementation, the benchmark, and a description of the hardware. They could use PyTorch, Triton, and AITER kernels, but no serving code from SGLang, vLLM, or TensorRT-LLM. They wrote the scheduler, caches, parallelism, graph capture, and several kernels themselves (~19.4k lines). (3/n)
1
47
The objective was peak goodput: throughput of responses that meet the latency SLO (p95 TPOT ≤ 250 ms, p95 TTFT ≤ 10 s) on a multi-turn chat workload that ramps from 1 to 192 concurrent sessions. (2/n)
8
🔥 Optimizing LLM serving requires repeated experiments, and running them on real hardware is slow and costly. Performance depends on how models, kernels, hardware, and serving policies interact. As models and workloads change, engineers must revisit their configuration choices. 🚀 Introducing ServingStudio: an integrated workbench for simulating, analyzing, and optimizing LLM serving systems. 🧵 More in the thread below. (1/9)
2
14
7
29
7,189
more replies
🔭 We plan to: - Support newer models, including GLM-5.3 Flash, DeepSeek V4.1 Flash, and Kimi K3. - Model distributed prefix caches, including cache offloading across GPU memory, host memory, and remote storage. - Enable remote hardware profiling and release a public kernel-performance database. (8/9)
2
1
349
Conclusion ServingStudio Sim helps engineers explore optimization ideas faster in simulation. ServingStudio Agent then implements promising changes and validates their effects on real hardware. 🤝 Explore ServingStudio and try it with your workload. 🌐 Website: syfi-servingstudio.github.io… 🛠️ Code: github.com/SyFI-ServingStudi… This work was led by @KanZhu854772, @MichaelGu341332, Sheetal Sriram, @mat_jacob1002, @KeisukeKamahori, @sudopowr, @conlesspan, @DedongX, @thepadawang, @arvind_uw, and @bariskasikci. (9/9)
1
405
Load more