
👀
"Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding"
Speculative decoding for RL rollouts!
This paper speeds up post-training without changing the target policy’s sampling distribution. So a draft model proposes multiple tokens, and the policy model verifies them.
An important system piece is that the draft must stay aligned with the continually updated policy, with weight sync and optional online adaptation.
This gives faster rollouts, but same learning trajectory. With ~1.5-1.8x rollout speedup at 8B, and projected ~2.5x end-to-end speedup at 235B scale.
2
675
Hayate Iso retweeted
Checkout how @vllm_project accelerate NeMo-RL rollout ⬇️
RL post-training is hitting a rollout bottleneck.
This new paper from #NVIDIAResearch shows how speculative decoding in NeMo-RL + @vllm_project can accelerate rollouts losslessly, with 1.8x higher throughput at 8B and projected 2.5x end-to-end speedup at 235B.
Read the full paper: nvda.ws/49kX9eo
Our new paper on RL acceleration with speculative decoding is out. You can try it today in NeMo RL v0.6 :)
docs.nvidia.com/nemo/rl/late…
RL post-training is hitting a rollout bottleneck.
This new paper from #NVIDIAResearch shows how speculative decoding in NeMo-RL + @vllm_project can accelerate rollouts losslessly, with 1.8x higher throughput at 8B and projected 2.5x end-to-end speedup at 235B.
Read the full paper: nvda.ws/49kX9eo
6
419
Hayate Iso retweeted
Announcing NVIDIA Nemotron 3 Super!
💚120B-12A Hybrid SSM Latent MoE, designed for Blackwell
💚36 on AAIndex v4
💚up to 2.2X faster than GPT-OSS-120B in FP4
💚Open data, open recipe, open weights
Models, Tech report, etc. here:
research.nvidia.com/labs/nem…
And yes, Ultra is coming!
Hayate Iso retweeted
What if you could ask a chatbot a question the size of an entire encyclopedia—and get an answer in real time?
Multi-million token queries with 32x more users are now possible with Helix Parallelism, an innovation by #NVIDIAResearch that drives inference at huge scale.
🔗 nvda.ws/4eCXxqh
Hayate Iso retweeted
Inference at scale is pushing the boundaries of what today’s systems can handle. One promising direction? #DisaggregatedInference — splitting the serving pipeline into distinct stages like prefill and decode — to unlock better performance across the throughput-interactivity spectrum.
While the concept has gained traction and sparked a wave of open source experimentation, real-world deployments remain rare. Why? Because the optimization space is vast, and orchestrating system-level coordination is incredibly complex.
📄 In our latest #NVIDIAResearch, we present a comprehensive systematic study of disaggregated inference at scale, evaluating hundreds of thousands of design points across varied model sizes, traffic patterns, and hardware configurations.
Key findings:
1️⃣ Disaggregation shines in prefill-heavy traffic patterns and with larger models.
2️⃣ Dynamic rate matching and elastic scaling are essential to achieving Pareto-optimal performance.
3️⃣ There's no one-size-fits-all architecture — workload-aware tuning is critical.
Whether you're building production inference infrastructure or exploring #LLM optimization techniques, this work offers actionable insights to balance system throughput with low-latency responsiveness.
🔬 Read the paper and dive into the data — let’s shape the future of scalable #AI serving, together.
📗research.nvidia.com/publicat…
Hayate Iso retweeted
👀New #NVIDIAResearch on boosting MoE model performance with disaggregated serving.
Learn how our NVIDIA Dynamo and GB200 NVL72 work together to boost the performance of #AI data centers running MOE models like DeepSeek R1 and the new Llama 4. ⚡
Technical deep dive➡️ nvda.ws/45eElfW
Hayate Iso retweeted
🚀 Want to improve your LLM responses? Read our tutorial for implementing AmbigNLG! Addressing task ambiguity in Natural Language Generation to drive more accurate, context-aligned outputs. @iso_map
megagon.ai/ambignlg-a-tutori… #AmbigNLG #NLP #tutorial #LLMs #EMNLP2024 #MLEngineering
🌴Heading to #EMNLP2024! Presenting AmbigNLG with @ayaniwa1213 on Tuesday at 4pm (Riverfront Hall).
Paper: aclanthology.org/2024.emnlp-…
Data: github.com/megagonlabs/ambig…
You can also stop by our @MegagonLabs sponsor booth or DM me to chat about full-time and internship opportunities :)
Hayate Iso retweeted
The CFP of DAIS@ICDE2025 has been released 👇
The first call for papers of DAIS 2025 is out.
👉Submission: January 20, 2025, 11:59pm Pacific Time
👉Notification: February 20, 2025
👉Camera-ready due: March 20, 2025
Please visit the website for more details: dais-workshop-icde.github.io…
Hayate Iso retweeted
📢Excited to bring the DAIS workshop to ICDE'25 (w/ @SainyamGalhotra @FarihaAnna @MikeCafarella @sairamgv) The focus is on the emerging idea of compound AI systems with a specific emphasis on data discovery, interactions w/ data, architectures for #AgenticAI+#LLM, and evaluation.
We are excited to announce the First Workshop on Data-AI Systems@ICDE 2025 (DAIS). The workshop will focus on exploring innovative approaches towards building data-aware compound AI systems in the #LLM era. More details: dais-workshop-icde.github.io…
#DataManagement #AgenticAI
Introducing 🫧 HoloBench—a new benchmark for measuring LLMs' reasoning capabilities across massive document collections!
- RAG models fetch info well but struggle with multi-doc reasoning.
- HoloBench evaluates how LLMs synthesize and aggregate info to solve complex tasks.
🧵 1/
We hope our work sets a new standard for evaluating holistic reasoning in LLMs! Proudly co-authored with @SAYg_7 (equal first author!) and Nikita Bhutani at @MegagonLabs.
💻 Code: github.com/megagonlabs/holob…
🗂 Dataset: hf.co/datasets/megagonlabs/h…
6/
Also paper link 🫠
🗒️ arxiv.org/abs/2410.11996
7/
150
Hayate Iso retweeted
🎉 Our long paper has been accepted for the #EMNLP2024 main conference!
In "AmbigNLG," co-authored with @iso_map, we tackle task ambiguity in NLG instructions to better align LLM outputs with your expectations.
📃 Read more: arxiv.org/abs/2402.17717

















