Research at NVIDIA | Led OmniVinci, MM-Ego, X-VILA | Contributed Nemotron-Omni
Santa Clara
Joined October 2012
- Tweets422
- Following2K
- Followers1K
- Likes1.6K
Pinned Tweet
@NVIDIA introduces Nemotron 3 Nano Omni LLMs.
A year ago, we started research on omni-modal LLMs—covering everything from architecture to data—and released OmniVinci (ICLR 2026). We’ve received some feedback since then, but two clear themes emerged: 9B parameters weren't quite enough for many applications, and restrictive licensing was holding back commercial use.
Therefore, we’ve teamed up with a larger group of amazing NVIDIA researchers/engineers to introduce NVIDIA's first commercial omni-LLM Nemotron 3 Nano Omni. This new 30B-A3B model is fully open-source and commercially usable. It also consistently outperforms similar-sized models across image, video, audio, and omni benchmarks.
My primary focus on this project was Omni-RL infra development and model training, besides architecture/hyper-parameter exploration and verification. Here are a few key takeaways from the process:
• Cross-modal synergy is real: In multimodal RL, improvements in one modality reinforce the others, translating directly into stronger overall benchmark scores.
• Data quality beats quantity: When it comes to RL, a steep reward curve slope is often much more critical than sheer training duration.
• Untapped potential in RL infra/algorithm: Especially for multimodal setups, there is still so much room to explore and optimize underlying architectures and scheduling systems.
To better support both the research community and industry developers, we aren't just releasing the model weights—we are also open-sourcing the training datasets and training code.
All LLMs should be omni-modality. We are looking forward to hearing your thoughts and feedback!
Blog: nvda.ws/4tCmhFS
Hugging Face: nvda.ws/4u79ue9
Tech Report: nvda.ws/4tDzHBq
#AI #MultimodalAI #OpenSource #ReinforcementLearning
Hanrong YE @ Nvidia retweeted
Is on-policy distillation really necessary? 🤔
📄 arxiv.org/pdf/2610.11291
🚀 Semi-OPD—distilling on fixed initial-student rollouts—beats OPD on 14/17 teacher–student pairs, with up to +13.6% accuracy and 11.4× training speedup.
More on-policy ≠ better. Alignment matters.
This is really something
Exclusive: A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research reut.rs/4gJ7FPG
There can be no intelligence without scaling, regardless of which methodology you believe will ultimately lead to AGI.
Congrats @Mononofu for open-sourcing nothing 😀
I’m so excited that @JensenHuang is a believer in open source now, looking forward to the CUDA and GPU driver open source release!
Hanrong YE @ Nvidia retweeted
At @NVIDIAAI we continue to push open data, techniques and models forward because we know that every organization needs the freedom to build and deploy AI in their own way.
We're now the biggest institutional contributor on HuggingFace and we expect to continue publishing.
It's not charity or a science project - we know that when AI grows, NVIDIA's opportunities also grow.
More analysis on the state of open source AI here: aiworld.eu/story/the-new-eng…
Hanrong YE @ Nvidia retweeted
I was laid off by Amazon AGI today, along with many of my colleagues.
My job was deciding which data points matter for pretraining. Turns out I was the one that got filtered out.
At 1pm, the reminder for our weekly project sync still went off — for a project that no longer has a team. We could only exchange a smile that needed no words.
More seriously: everything I've worked on has been about data, data curation pipeline, data reordering, and data uncertainty. I’m actively looking for new opportunities, including pretraining, trustworthy LLMs, safety, and LLM efficiency — please reach out if you have any openings!
To my manager and teammates: thank you for all the help and support you've given me along the way — I learned so much, and I loved the work we did together. It was a great journey, and I know even better ones are ahead for each of you.
True
Why can Kimi ship K3? Let me tell my story.
Earlier this year, I left academia for industry. I talked to a lot of companies along the way. Here's what I saw:
1⃣Arrogance. They believe the AI war is over, and they won. No hunger for the future, and no hunger for talent.
2⃣Restlessness. Young labs short on foundation, either rushing to catch the frontier or pivoting away from the competition.
3⃣Fear. Strong teams with real experience, but from the second tier, they can't quite bring themselves to aim for #1.
4⃣Misalignment. Everyone is optimizing for their own credit, but nobody really cares whether the company can reach AGI.
Kimi was different.
Over many conversations with the founders, the same thing came through every time: a raw, genuine hunger for AGI.
I joined. The hunger was real.
We shipped K3. This is only the beginning.
Well, if it’s useful, there must be some intelligence in it :)
Visual information density is definitely much lower than language, which makes visual generation far more difficult at our current compute scale.
We need more compute 😃
TL;DR: Our DLLM delivers higher image generation quality on popular benchmarks like GenEval compared to state-of-the-art models such as EMU3.5, while also achieving 40x faster inference speed. Check our Nemotron-Labs-Diffusion-Image paper!
@NVIDIAAI
Hanrong YE @ Nvidia retweeted
No more endless chasing AudioLM updates! 🏹🔊 Speech-Hands Agent has been accepted to @aclmeeting for oral & open-sourced, with a 5.69% WER as Voice-Agent and 77.37% Acc. on @DCASE_Challenge 25 AudioQA dev-set. Check our demo and open-claw PR 🦞arxiv.org/pdf/2601.09413
a joint work with Zhen Wan @ZhehuaiC @leoyerrrr @goelarushi27 @MXzBFhjFpS1jyMI @shinjiw_at_cmu @SreyanG @RafaelValleArt @RHachiuma @hirota_yusuke et al.
Hanrong YE @ Nvidia retweeted
We’re releasing Nemotron-Labs-Diffusion - the first Tri-mode LM family (3B/8B/14B) that switches between 1⃣Autoregressive, 2⃣Diffusion, and 3⃣Self-Speculation decoding by simply changing the attention pattern/mask.
One model Three decoding modes. No extra draft models. No architecture changes. Just significantly better efficiency across different concurrency levels.
Up to 4× higher real throughput for a single user.
🤗 HF Collection: huggingface.co/collections/n…, open license
🛜 Project page: research.nvidia.com/publicat…
📰 Tech report: bit.ly/Nemotron-Labs-Diffusi…
Details below 👇
Nemotron Omni has officially surpassed 1 million downloads on Hugging Face. Stay tuned for more omni models from NVIDIA!
Just wrapped up the latest model post-training right before ICLR, and finally getting to enjoy a rare relaxing weekend. 😃
Next stop: Brazil🇧🇷! I’ll be presenting OmniVinci, the 9B audio-video omni-modal understanding research that was also my first project at NVIDIA.
Hanrong YE @ Nvidia retweeted
🚨 Now live as CVPR 2026 Highlight!!!
R4D-Bench dataset + official NVlabs GitHub just dropped.
1K+ region-level 4D VQA pairs ready to download today!
Thread + links👇
#CVPR2026 #4DRGPT #R4DBench
Current Vision-Language Models completely struggle with complex 4D dynamics. We fixed that. 🤯
🚨 Introducing 4D-RGPT: distilling perceptual knowledge directly into LLMs for precise space & time reasoning.
🎉 Excited to share our @NVIDIAAI work has been accepted to #CVPR2026! @CVPR
A quick dive into how it works 🧵👇
Hanrong YE @ Nvidia retweeted
Humans can see in high-res, high-FPS in real-time. Why can't VLMs?
Introducing AutoGaze: ViTs/VLMs "gaze" only at key video regions! Up to 4-100x token savings, 19x speedup, and enables scaling to 4K-res 1K-frame videos.
📄 arxiv.org/abs/2603.12254
🌐 autogaze.github.io
🤗 huggingface.co/collections/b…
(1/n)🧵