Rui Pan 潘瑞 (@ruipeterpan) header
Rui Pan 潘瑞 (@ruipeterpan) profile picture

Member of Technical Staff @FireworksAI_HQ, efficient LLM inference, previously PhD @PrincetonCS @Google @AmazonScience @maxplanckpress @WisconsinCS

Princeton, NJ
Joined August 2016
Photo by Rui Pan 潘瑞 (@ruipeterpan)Photo by Rui Pan 潘瑞 (@ruipeterpan)Photo by Rui Pan 潘瑞 (@ruipeterpan)Photo by Rui Pan 潘瑞 (@ruipeterpan)Photo by Rui Pan 潘瑞 (@ruipeterpan)Photo by Rui Pan 潘瑞 (@ruipeterpan)
Rui Pan 潘瑞 retweeted
Detaining a PhD student on her way back to Princeton from presenting scientific work at a conference is not how we get the best and brightest scientists in the world to want to come to the US. I hope this is resolved quickly! More details: dailyprincetonian.com/articl…
Our colleague Eva Yi Xie, a Princeton neuroscience PhD student, was detained by ICE at Reagan National Airport on her way home from SfN, where she’d just presented her poster. She’s being held in Chantilly, VA. Please help us bring her home. 📷
399
661
53
6,871
537,881
Rui Pan 潘瑞 retweeted
This is stupid. What counts as a chinese model? What about Thinky's model distilled from Kimi? What about composer, continued pretraining? What about Nemotron, using tons of DeepSeek synthetic data? What about that silly nous finetune for "american" values?
SITUATION DETECTED: The White House and Treasury are discussing a proposal that would require US companies to publicly disclose the use of Chinese open-weight AI models on their websites, aiming to discourage their use through public pressure, per Reuters.
39
125
23
1,545
75,487
Same
I signed the letter (standwithjensen.com). It’s touching to see overwhelming support from people who actually work in tech and AI, and thus understand and appreciate the value of Jensen. Most people who are eager to remove Jensen seem to share the following reasons: 1) Not understanding how tech innovation works, let alone AI. I can understand that some people believe management and governance leads to true innovation; however, it's never the case 2) On US-China AI race, not understanding China's AI industry and ecosystem 3) Believing unsubstantiated allegations and conspiracy theories 4) Other purposes, e.g., political propaganda (but that does not help the country) 5) Condemning his moral conduct. I can relate to and understand people who feel this way (but that's not the reason to immediately remove him from the company)
6
10
1
75
7,918
Rui Pan 潘瑞 retweeted
In March, I emailed Jensen when @inferact's compute provider took back our capacity, leaving us unable to develop @vllm_project for Blackwell. He replied within the SAME minute and had the issue resolved overnight. The entire NVIDIA team, led by Jensen, operates like an unstoppable armada. Jensen moves fast, so is his "exo-skeleton". Everyone in the industry respects and trusts him. That's how you push open intelligence forward. Thank you Jensen, for saving Inferact and vLLM.
In June @NaderLikeLadder emailed Jensen about a crazy idea we had over dinner to make Local AI 10x better on NVIDIA hardware. Jensen replied in 7 minutes. He saw this as important work, and approved a war room for us to work on this at NVIDIA HQ. For a month, NVIDIA HQ was the @exolabs satellite office. We were overwhelmed by the support we got. We were back to back the first week with smart NVIDIANs coming in to our war room to help us. Creators of CUDA, Dynamo, people who built NVIDIA. Long story short with NVIDIA we managed to improve their Local AI playbooks by 9.6x (almost the 10x goal we set out with). We’ll be publishing a whitepaper on this work. We were just a small startup with a crazy idea. Jensen took a bet on us. Jensen cares deeply about Open Source AI and Local AI. I don't believe Jensen is replaceable, he's a 1 of 1 human, but I hope NVIDIA will continue to aggressively support Open Source AI and Local AI, and continue to take bets on small startups with crazy ideas. Thank you Jensen
31
317
30
4,516
208,030
Rui Pan 潘瑞 retweeted
4
31
11
436
22,900
Rui Pan 潘瑞 retweeted
Does it matter where you run an open-weight model? Today we are releasing the Inference Provider Index. It compares first and third-party open-weight models for quality on agentic tasks in finance, coding, and legal industries. Unlike other inference benchmarks, we rerun it every two weeks to catch quality degradation, and we use agentic tasks, not single-turn multiple-choice questions. It covers over 650 hours of model runtime.
17
22
13
251
21,649
Rui Pan 潘瑞 retweeted
ReplaySSM now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang! It speeds up standard decoding (up to 1.47x) and speculative decoding (up to 2.1x), and now also enables a low-precision SSM state whose rounding error never propagates through the recurrence. Building on our earlier ReplaySSM work, we collaborated with @NVIDIA to pin down what the method really unlocks. Caching SSM inputs instead of the state is what ReplaySSM does. What it changes is that computing the output and updating the state can be separated during decoding. All three gains follow from that separation. Joint blog post, paper, and code with @NVIDIA and @tri_dao 👇
20
77
13
455
72,830
Rui Pan 潘瑞 retweeted
We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs. No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase. Meet Rigel 🧵
23
79
18
635
87,301
Rui Pan 潘瑞 retweeted
🚀Introducing Motus, the open-source agent infrastructure that learns in production. Existing agent infra serves static agents: the harness, model, and workflow are fixed after deployment. But static agents degrade over time. The harness goes stale, new models go unincorporated, context drifts, and latency compounds. Motus closes this gap by learning from every trace (failures, latency, cost, and task outcomes) and using those signals to continuously optimize agent harness, model orchestration, context memory, and end-to-end latency. Early results: higher accuracy than any single frontier model at 2.3× lower cost (Terminal-Bench 2.0, SWE-bench Verified), with 52% lower latency and 45% better memory recall. Open source under Apache 2.0. Works with any agent SDK. Deploy with one command. github.com/lithos-ai/motus lithosai.com/
22
71
12
570
59,349
Rui Pan 潘瑞 retweeted
New hierarchical memory optimization out! This time for sparse attention! Try it out and stay tuned for more Hi-series : )
🚀 New blog is out: HiSparse — Turbocharging Sparse Attention with Hierarchical Memory! Sparse attention cuts compute costs, but the full KV cache still sits in GPU HBM, making it capacity-bound. HiSparse fixes this. Results: ⚡️ 3× throughput at 256 concurrent requests vs. baseline (32K input, 8K output on 8×H200) 🚀 Up to 5× throughput on long-context scenarios (two H20 PD-disaggregated deployment) Key techniques include: 💾 Proactively offloads inactive KV cache to host memory, freeing GPU HBM for larger batch sizes 🧠 Hot device buffer keeps frequently accessed KV regions on-device to minimize swap-in latency 🔧 Custom CUDA kernel: top-k miss detection + LRU eviction + page table updates in one pass Currently supports DeepSeek Sparse Attention (DSA) models: DeepSeek-V3.2 and GLM-5.1. Thanks to @Zhiqiang_Xie and the team for this great contribution!
1
17
69
6,836
congrats to llama 3 large for winning the LLM trading contest by not participating
94
125
66
3,580
281,429
[1/N] 🚀 Excited to introduce my first work at @Princeton: LessIsMore – a training-free sparse attention method tailored for efficient reasoning in LRMs, achieving lossless accuracy with high sparsity up to 87.5% and 1.1x avg decoding speedup compared to Full Attention on reasoning tasks like AIME-24. (More details in 🧵) 💻 Code: github.com/DerrickYLJ/LessIs… 📄 arXiv: huggingface.co/papers/2508.0… 🔍 HF Daily Paper: huggingface.co/papers/2508.0…
8
11
3
54
8,710
Rui Pan 潘瑞 retweeted
With RAG and agents becoming ubiquitous in LLM systems, tuning quality and performance JOINTLY is essential to achieve the best LLM quality-of-experience. Our paper at SOSP this year, addresses this exact tradeoff!🔥
1
6
1
17
2,303
Rui Pan 潘瑞 retweeted
committed to doing my part in decreasing reviewer workload by writing fewer papers
8
18
2
234
15,627
Rui Pan 潘瑞 retweeted
A quick update on the future of the `transformers` library! In order to provide a source of truth for all models, we are working with the rest of the ecosystem to make the modeling code the standard. A joint effort with vLLM, LlamaCPP, SGLang, Mlx, Qwen, Glm, Unsloth, Axoloth, Deepspeed, IBM, Gemma, Llama, Deepseek, microsoft, nvidia, internLM, Llava, AllenAI, Cohere, TogetherAI.....
24
102
32
1,018
177,692
Rui Pan 潘瑞 retweeted
Congrats to @ruipeterpan and the entire team for the recognition of Marconi at MLSys. Hybrid mamba-transformer LLMs are getting more popular, and the change in compute / cache size ratio changes the way you should cache and schedule requests
Replying to @ruipeterpan
Our work received an honorable mention for the outstanding paper!!! 🤓
8
2
41
7,958
Rui Pan 潘瑞 retweeted
I will be presenting our work, Decoding Speculative Decoding, at @naacl tomorrow. We identified the performance bottleneck in speculative decoding to be draft model depth and demonstrated low correlation between language modeling performance and token acceptance rate.
1
2
11
988
Rui Pan 潘瑞 retweeted
Tomorrow at 3:50pm, Neil (Ravi’s PhD at Princeton / my ex-intern at MSR) will be presenting 𝙈𝙤𝙬𝙜𝙡𝙞, a real-time video rate control algorithm learned solely from log data — in the same spirit as “Learning in situ” (NSDI ’20). Intrigued? Join us to learn more at NSDI ’25!
1
2
23
1,707
Rui Pan 潘瑞 retweeted
🚀 Introducing Hunyuan-TurboS – the first ultra-large Hybrid-Transformer-Mamba MoE model! Traditional pure Transformer models struggle with long-text training and inference due to O(N²) complexity and KV-Cache issues. Hunyuan-TurboS combines: ✅ Mamba's efficient long-sequence processing ✅ Transformer's strong contextual understanding 🔥 Results: - Outperforms GPT-4o-0806, DeepSeek-V3, and open-source models on Math, Reasoning, and Alignment - Competitive on Knowledge, including MMLU-Pro 1/7 lower inference cost than our previous Turbo model 📌 Post-Training Enhancements: - Slow-thinking integration improves math, coding, and reasoning - Refined instruction tuning boosts alignment and agent execution - English training optimization for better general performance 🎯 Upgraded Reward System: - Rule-based scoring & consistency verification - Code sandbox feedback for higher STEM accuracy - Generative-based reward improve QA and creativity, reducing reward hacking The future of AI is here! 🚀
67
205
82
1,189
1,273,301