Hayate Iso (@iso_map) profile picture
Santa Clara, CA
Joined September 2023
Photo by Hayate Iso (@iso_map)Photo by Hayate Iso (@iso_map)Photo by Hayate Iso (@iso_map)Photo by Hayate Iso (@iso_map)Photo by Hayate Iso (@iso_map)Photo by Hayate Iso (@iso_map)Photo by Hayate Iso (@iso_map)Photo by Hayate Iso (@iso_map)
👀
"Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding" Speculative decoding for RL rollouts! This paper speeds up post-training without changing the target policy’s sampling distribution. So a draft model proposes multiple tokens, and the policy model verifies them. An important system piece is that the draft must stay aligned with the continually updated policy, with weight sync and optional online adaptation. This gives faster rollouts, but same learning trajectory. With ~1.5-1.8x rollout speedup at 8B, and projected ~2.5x end-to-end speedup at 235B scale.
2
675
Hayate Iso retweeted
Checkout how @vllm_project accelerate NeMo-RL rollout ⬇️
RL post-training is hitting a rollout bottleneck. This new paper from #NVIDIAResearch shows how speculative decoding in NeMo-RL + @vllm_project can accelerate rollouts losslessly, with 1.8x higher throughput at 8B and projected 2.5x end-to-end speedup at 235B. Read the full paper: nvda.ws/49kX9eo
1
9
1
33
5,518
Our new paper on RL acceleration with speculative decoding is out. You can try it today in NeMo RL v0.6 :) docs.nvidia.com/nemo/rl/late…
RL post-training is hitting a rollout bottleneck. This new paper from #NVIDIAResearch shows how speculative decoding in NeMo-RL + @vllm_project can accelerate rollouts losslessly, with 1.8x higher throughput at 8B and projected 2.5x end-to-end speedup at 235B. Read the full paper: nvda.ws/49kX9eo
6
419
Hayate Iso retweeted
Announcing NVIDIA Nemotron 3 Super! 💚120B-12A Hybrid SSM Latent MoE, designed for Blackwell 💚36 on AAIndex v4 💚up to 2.2X faster than GPT-OSS-120B in FP4 💚Open data, open recipe, open weights Models, Tech report, etc. here: research.nvidia.com/labs/nem… And yes, Ultra is coming!
61
203
49
1,201
212,126
What if you could ask a chatbot a question the size of an entire encyclopedia—and get an answer in real time? Multi-million token queries with 32x more users are now possible with Helix Parallelism, an innovation by #NVIDIAResearch that drives inference at huge scale. 🔗 nvda.ws/4eCXxqh
4
28
8
141
17,094
Inference at scale is pushing the boundaries of what today’s systems can handle. One promising direction? #DisaggregatedInference — splitting the serving pipeline into distinct stages like prefill and decode — to unlock better performance across the throughput-interactivity spectrum. While the concept has gained traction and sparked a wave of open source experimentation, real-world deployments remain rare. Why? Because the optimization space is vast, and orchestrating system-level coordination is incredibly complex. 📄 In our latest #NVIDIAResearch, we present a comprehensive systematic study of disaggregated inference at scale, evaluating hundreds of thousands of design points across varied model sizes, traffic patterns, and hardware configurations. Key findings: 1️⃣ Disaggregation shines in prefill-heavy traffic patterns and with larger models. 2️⃣ Dynamic rate matching and elastic scaling are essential to achieving Pareto-optimal performance. 3️⃣ There's no one-size-fits-all architecture — workload-aware tuning is critical. Whether you're building production inference infrastructure or exploring #LLM optimization techniques, this work offers actionable insights to balance system throughput with low-latency responsiveness. 🔬 Read the paper and dive into the data — let’s shape the future of scalable #AI serving, together. 📗research.nvidia.com/publicat…
4
22
2,385
👀New #NVIDIAResearch on boosting MoE model performance with disaggregated serving. Learn how our NVIDIA Dynamo and GB200 NVL72 work together to boost the performance of #AI data centers running MOE models like DeepSeek R1 and the new Llama 4. ⚡ Technical deep dive➡️ nvda.ws/45eElfW
7
15
2
58
5,411
🌴Heading to #EMNLP2024! Presenting AmbigNLG with @ayaniwa1213 on Tuesday at 4pm (Riverfront Hall). Paper: aclanthology.org/2024.emnlp-… Data: github.com/megagonlabs/ambig… You can also stop by our @MegagonLabs sponsor booth or DM me to chat about full-time and internship opportunities :)
4
1
28
7,878
The CFP of DAIS@ICDE2025 has been released 👇
The first call for papers of DAIS 2025 is out. 👉Submission: January 20, 2025, 11:59pm Pacific Time 👉Notification: February 20, 2025 👉Camera-ready due: March 20, 2025 Please visit the website for more details: dais-workshop-icde.github.io…
3
4
794
📢Excited to bring the DAIS workshop to ICDE'25 (w/ @SainyamGalhotra @FarihaAnna @MikeCafarella @sairamgv) The focus is on the emerging idea of compound AI systems with a specific emphasis on data discovery, interactions w/ data, architectures for #AgenticAI+#LLM, and evaluation.
We are excited to announce the First Workshop on Data-AI Systems@ICDE 2025 (DAIS). The workshop will focus on exploring innovative approaches towards building data-aware compound AI systems in the #LLM era. More details: dais-workshop-icde.github.io… #DataManagement #AgenticAI
5
9
1,030
Introducing 🫧 HoloBench—a new benchmark for measuring LLMs' reasoning capabilities across massive document collections! - RAG models fetch info well but struggle with multi-doc reasoning. - HoloBench evaluates how LLMs synthesize and aggregate info to solve complex tasks. 🧵 1/
1
1
3
723
⚡️Are you working with #NLP or #AI-driven products for Natural Language Generation (#NLG)? Task ambiguity is a common pain point and AmbigNLG changes that! AmbigNLG is designed to solve task ambiguity in instructions for NLG. What it is... 👇🧵
1
2
5
599
🎉 Our long paper has been accepted for the #EMNLP2024 main conference! In "AmbigNLG," co-authored with @iso_map, we tackle task ambiguity in NLG instructions to better align LLM outputs with your expectations. 📃 Read more: arxiv.org/abs/2402.17717
5
10
121
9,653