
building agents and poses @nvidia | Robograd Biorobotics lab @CMU_Robotics | Investing in startups | Taking side of folks who build
San Francisco, CA
Joined June 2009
- Tweets1.4K
- Following3.3K
- Followers1.1K
- Likes38.7K
Pinned Tweet
Check the full suite for full mocap to robotics pretraining . SOMA has anatomically correct joint definitions and has much detailed mesh key points compared to MHR/SMPL. Foundational for all bodypose downstream tasks. More on this soon on its capabilities.
#NVIDIA just released a whole ecosystem for human(oid) motion and robot learning from human data. 🚀🦾
Data, as we all know, is the key to scaling AI models. To accelerate the field of Embodied AI, we have open-sourced a full stack of models and tools to capture, generate, retarget, and simulate human(oid) motion data at scale, along with a massive high-quality dataset and a standard human skeletal representation, SOMA, to make them all seamlessly communicate with each other.
The entire suite is available under the Apache 2.0 license.
1️⃣ SOMA: A universal interface to unify all parametric human body models (SOMA-shape, SMPL, MHR, etc.) into a standard skeletal representation, eliminating the need for custom adapters or model-specific retargeting.
🔗 lnkd.in/gsxhiJnn
2️⃣ Kimodo: High-fidelity, controllable text-to-motion generation for both humans and humanoid robots.
🔗 lnkd.in/gCc84XnX
3️⃣ GEM: A global human pose estimation method from in-the-wild videos, natively compatible with SOMA.
🔗 lnkd.in/g_QAvRjn
4️⃣ Bones-SEED: A massive dataset of 150k+ motions in SOMA format, including data already retargeted for the Unitree G1, created with our partners at Bones Studio.
🔗 lnkd.in/gfx-QD-w
🔗 lnkd.in/gyNdTwQx
5️⃣ SOMA Retargeter: A dedicated tool for seamless motion retargeting from the SOMA skeleton to the Unitree G1.
🔗 lnkd.in/gqz9Na-H
6️⃣ ProtoMotions: Our high-performance simulation framework for training digital human(oid)s via RL, now with native SOMA support.
🔗 lnkd.in/gmvMikMU
This is just the beginning, and we have much more in the pipeline. Excited to see what the community builds next!
#NVIDIA #GTC #GTC2026 #Robotics #EmbodiedAI #PhysicalAI @NVIDIAAI
Today, we're sharing a major milestone in our mission to democratize frontier AI development:
Pretraining at scale without a pretraining team.
We set out to answer an ambitious question: Can we significantly reduce the time, human expertise, and compute needed to pretrain a competitive foundation model from scratch?
The result is Mirendil-Base, our first pretrained coding model, built by one researcher in just 3.5 months with less than $10M in compute, including all research, experimentation, and the final training run.
Our approach is already showing promising results as we scale to even larger models.
This is an important step toward a future where a single researcher, empowered by AI, can accomplish what once required an entire research organization. As these capabilities improve, we believe frontier AI development can become accessible to far more people and teams.
More to come soon, including independent evaluations, post-training benchmarks, and API access.
Harsh retweeted
Ask ChatGPT for a new scene, and Swap Anything keeps every movement while changing everything else.
Harsh retweeted
Nvidia is the now the most active VC investor in the world, and the largest in Europe.
Crazy
cc @dealroomco
BREAKING: Jensen has just announced that @nvidia in investing $1bn+ into 5 UK startups:
>> Boltz which is building biomolecular models and AI agents that helps scientists predict molecular structures and interactions
>> Foresight which helps owners of major infrastructure projects predict completion dates, mitigate risks and accelerate time to revenue.
>> Hiverge which is building an AI discovery engine that automatically writes and optimizes algorithms for tasks such as faster AI training and complex logistics planning.
>> Prima Mente which focuses on building biological foundation models to decode complex diseases and identify novel therapeutic targets, starting with Alzheimer’s, Parkinson’s and ALS.
>> Riverlane which builds real-time error correction hardware and software that work across all major qubit types, helping quantum computers become reliable enough for practical use.
He announced the news in London this evening.
Apologies for the terrible pic which includes @rowlsmanthorpe’s side profile
Harsh retweeted
NVIDIA just released Kimodo, a controllable human motion diffusion model, on Hugging Face
It turns text and kinematic constraints into high-quality 3D human and robot motion, trained on 700 hours of mocap data.
Harsh retweeted
A rare post that takes <10 seconds to read but an hour to understand the depth of it.
Tom has written an Econ koan.
Harsh retweeted
Nemotron-Labs models give us a glimpse of what the future might be. And it may be looped!
Dory1.0-4B is an impressive 4B model!
huggingface.co/nvidia/NVIDIA…
Harsh retweeted
We are excited to announce that we post-trained NVIDIA Nemotron-3.5-Super-VL to be an autonomous ML researcher. We show that our post-trained model goes from 28% to 35% on AutoResearchExam, matching Qwen 3.8 Max, a model with 20× the total parameters.
The tasks are long horizon, open ended AI and ML research: building a faster executor for a transformer compute graph, training a RL control policy, creating new uncertainty quantification methods, etc. The model gets to submit once, and it doesn't see its score until the end.
We curated SFT data using Self-context distillation. This means that we let the Nemotron model to solve a problem with hints, then we SFT on these trajectories but without the hints. This gave us +11% on AutoResearchExam.
As a next step, we added reflection: We ask the Nemotron model to review its own past attempts and writes concrete lessons learned. Then we SFT on these reflections.
Surprisingly, training on these lessons improved the performance to +25% compared to the base model.
The hardest part was choosing the right data. We selected traces based on task scores, completion checks, and runtime, then refined the sampling across training rounds to address failures in the base and fine-tuned models. We removed hints and filtered out steps that relied on information available only in those hints. The goal was to teach decisions supported by the task and history the model would actually see at inference.
Full write-up:
bespokelabs.ai/blog/teaching…
Thanks to @NVIDIAAI for early access and collaborating on this.
Harsh retweeted
We partnered with OpenAI and basically made an entire editor and all-in-one creative studio inside your ChatGPT.
Now your first successful generation with our Pika plugin on GPT gives you 250 bonus credits too! Install now pika.art/mcp/chatgpt/install…
Harsh retweeted
Yes, we launched TWO models today! ✨✨
* Nemotron-3.5-Super for your DGX Sparks and H100s
* Nemotron-3.5-Bolt for your RTX Sparks
huggingface.co/nvidia/NVIDIA…
huggingface.co/nvidia/NVIDIA…
Harsh retweeted
We have never seen a model do this before, but GLM-5.3 sometimes reasons (unprompted) about how its current actions could become training data and impact its future self. 🧵
Harsh retweeted
Run Nemotron 3.5 Super VL locally on 64GB Mac 💻
We released 7 quants of @NVIDIAAI's new 120B vision model on Hugging Face from 131.6GB down to 59.3GB
For a 64GB Mac we recommend the 59.3GB AD-IQ2_M with a top-1% of 89.8% compared to BF16
GGUFs 👇
huggingface.co/AtomicChat/NV…
Harsh retweeted
🎆 Nemotron-3.5-Super-VL is here! 🎆
✅ Same base architecture as Super-3.0: 120B total, 12B active plus Vision encoder
✅ Ahead of Ultra-3.0 on most benchmarks! And now with visual reasoning!
✅ SWEBench-Pro 51.79%, AA index of 26 vs 23 for Ultra-3.0
Post-training pipeline is a greatly simplified and improved version of what we had for Ultra-3.0.
We reduced number of teachers, SFT set size while significantly improving data and environments quality and made it multimodal! 1/5
🤖 Made with AI
its a great day for open models
Very excited to announce our new models: Nemotron 3.5 Super and Lightning!
This also feels like a good time to share that a few months ago I joined the amazing Nemotron Pretraining team. We're cooking some very large and very strong models for you!
Here is a tech summary for 3.5:
* Super is a 120B total / 12B active vision language model (VLM)
* Lightning is a 30B total / 3B active text-only LLM
* Super scores 51.78% on SWE-bench Pro (better than Nemotron 3 Ultra despite ~4.6× fewer active params!)
* arch: MoE Hybrid Mamba-Transformer designed for efficient inference, with a 1M token ctx window, and native speculative decoding (via MTP)
* training is done across our GB200 and GB300 (and deployment validated across all imaginable SKUs, including consumer (RTX) and DGX Spark)
A few details on how we train these:
-- Pretraining
* pipeline: initialize with Nemotron 3 base -> custom MTP initialization + 100B-token MTP warm-up -> 1T-token distillation from the next larger N3 teacher
* Lightning then goes through a 21B token long-context (LC) stage (@ 1M ctx)
* Super goes through:
- Indic CPT: 100B tokens at 8K (10B Indic + 90B replay, we built a synthetic corpus with 2.83T toks across 22 Indian langs! Indic benchmarks improve by up to 9.82%)
- LC extension: 35B tokens at 1M
- Agentic SWE mid-training: 65.4B tokens at 256K (uses whole repos, issue-PR pairs, commit histories, etc.); DeepSWE pass@1 improves from 6.4 -> 10.4
- LC recovery: 9B tokens at 1M (as we observed regression after mid-training)
-- Post-training
* Super: two-stage SFT -> joint RLVR/RLHF -> specialist teachers -> two-stage MOPD -> MTP boosting
* Lightning uses a lighter text-only pipeline: single-stage SFT -> RLVR
* Super's SFT transforms the text-only base into a VLM using vision, agentic, reasoning, chat, safety, LC data
* A few examples of RLVR environments:
- Olympiad-level competitive programming
- Finance agents using SEC search, document parsing, evidence retrieval, and tool use
* Specialist teachers include Agentic, Reasoning, Video reasoning and tracking, Vision tool-use, Vision-reasoning, and an Alignment/RLHF teacher
* Two-stage MOPD consolidates these specialist capabilities into one model
* Post-training ends with MTP boosting to better align the drafter and reduce recursive train–inference mismatch; this improves Super's SPEED-Bench acceptance length by 21.8% (to 4.612 tokens)
-- Inference
The model is highly optimized for inference (with plenty of quantized options to choose from)! A few data points:
* On a GB200, Super reaches 4,856 output toks/s/GPU with ≥50 toks/s/user on a 20-turn agentic workload
* Super provides 2.34x the throughput of Qwen3.5 122B-A10B and 1.90x the throughput of Qwen3.8 Flash-Next
* Native MTP is complemented by standalone parallel drafters: LiLiCorr for Super, and DFlash and DSpark for Lightning
Looking forward to seeing what you do with 3.5! Phase shift for Nemotron incoming! :)) (with the usual this will be our weakest model ever, but really really so)
paper: research.nvidia.com/labs/nem…
Harsh retweeted
Very excited to announce our new models: Nemotron 3.5 Super and Lightning!
This also feels like a good time to share that a few months ago I joined the amazing Nemotron Pretraining team. We're cooking some very large and very strong models for you!
Here is a tech summary for 3.5:
* Super is a 120B total / 12B active vision language model (VLM)
* Lightning is a 30B total / 3B active text-only LLM
* Super scores 51.78% on SWE-bench Pro (better than Nemotron 3 Ultra despite ~4.6× fewer active params!)
* arch: MoE Hybrid Mamba-Transformer designed for efficient inference, with a 1M token ctx window, and native speculative decoding (via MTP)
* training is done across our GB200 and GB300 (and deployment validated across all imaginable SKUs, including consumer (RTX) and DGX Spark)
A few details on how we train these:
-- Pretraining
* pipeline: initialize with Nemotron 3 base -> custom MTP initialization + 100B-token MTP warm-up -> 1T-token distillation from the next larger N3 teacher
* Lightning then goes through a 21B token long-context (LC) stage (@ 1M ctx)
* Super goes through:
- Indic CPT: 100B tokens at 8K (10B Indic + 90B replay, we built a synthetic corpus with 2.83T toks across 22 Indian langs! Indic benchmarks improve by up to 9.82%)
- LC extension: 35B tokens at 1M
- Agentic SWE mid-training: 65.4B tokens at 256K (uses whole repos, issue-PR pairs, commit histories, etc.); DeepSWE pass@1 improves from 6.4 -> 10.4
- LC recovery: 9B tokens at 1M (as we observed regression after mid-training)
-- Post-training
* Super: two-stage SFT -> joint RLVR/RLHF -> specialist teachers -> two-stage MOPD -> MTP boosting
* Lightning uses a lighter text-only pipeline: single-stage SFT -> RLVR
* Super's SFT transforms the text-only base into a VLM using vision, agentic, reasoning, chat, safety, LC data
* A few examples of RLVR environments:
- Olympiad-level competitive programming
- Finance agents using SEC search, document parsing, evidence retrieval, and tool use
* Specialist teachers include Agentic, Reasoning, Video reasoning and tracking, Vision tool-use, Vision-reasoning, and an Alignment/RLHF teacher
* Two-stage MOPD consolidates these specialist capabilities into one model
* Post-training ends with MTP boosting to better align the drafter and reduce recursive train–inference mismatch; this improves Super's SPEED-Bench acceptance length by 21.8% (to 4.612 tokens)
-- Inference
The model is highly optimized for inference (with plenty of quantized options to choose from)! A few data points:
* On a GB200, Super reaches 4,856 output toks/s/GPU with ≥50 toks/s/user on a 20-turn agentic workload
* Super provides 2.34x the throughput of Qwen3.5 122B-A10B and 1.90x the throughput of Qwen3.8 Flash-Next
* Native MTP is complemented by standalone parallel drafters: LiLiCorr for Super, and DFlash and DSpark for Lightning
Looking forward to seeing what you do with 3.5! Phase shift for Nemotron incoming! :)) (with the usual this will be our weakest model ever, but really really so)
paper: research.nvidia.com/labs/nem…
Harsh retweeted
Tonight Jensen met with President @EmmanuelMacron to discuss NVIDIA’s contribution to France’s thriving AI ecosystem through investments spanning startups, AI-native companies, and computing infrastructure.
They also covered AI safety and our recently launched Open Agent Safety Platform.
.@JensenHuang wrapped up his Paris visit at the Élysée with President @EmmanuelMacron, discussing NVIDIA’s investments in France’s AI ecosystem, open source, and AI safety.
Our road to #NVIDIAGTC Berlin continues.





































