Director of Robotic Systems @NVIDIA. Isaac Cortex, cobots, geometric methods; PhD CMU, research Max Planck, TTI-C, co-founder Lula Robotics, eng Google, Amazon
Seattle, WA
Joined December 2010
- Tweets565
- Following422
- Followers1.6K
- Likes3.4K
Nathan Ratliff retweeted
After we released the original Isaac Gym and then the Orbit framework, I took a step back from robotics to focus on 3D reconstruction and rendering - NeRFs and 3D Gaussians.
Then, in October 2024, I got a call from @JensenHuang , asking me to come back and take charge of Isaac Lab.
I was really enjoying being deep in the weeds working on technologies that were defining the future of 3D graphics. I also understood exactly what it would mean to take this on, and the level of commitment it would require.
So, I hesitated.
“It’s very flattering that you think I’m the right person for this, but I’ve been away from robotics for a couple of years, and things move fast. I don’t know our current strategy, or even who’s on the team now.”
Jensen’s response was simple, direct, and effective: “You decide.”
With that level of trust, the opportunity was hard to pass up.
Over the past two years, I’ve had the privilege of working with an extraordinary group of collaborators to define the future of robotics simulation with Newton and Isaac Lab 3.0.
I couldn’t be more proud of our incredible, hardworking team and what we’ve been able to accomplish together. Seeing everything come to fruition at GTC last week was amazing.
Thanks, Jensen!
We’re excited to announce the official release of NVIDIA Isaac Lab 3.0. 🎉
Isaac Lab 3.0 brings a new modular and lightweight architecture along with major new capabilities across the robot learning workflow: multi-physics and multi-rendering support, training at scale, and streamlined deployment on real hardware.
With Newton integration, the new OVRTX renderer, RL post-training, and LEAPP deployment, there’s a lot to explore.
Here’s a look at what’s new 🧵
(1/7)
Nathan Ratliff retweeted
I've been spending the last several months stress testing this software in beta for my research. It's so incredibly feature rich and flexible for all sorts of different needs and applications.
Critically, it is super scalable but also amenable to academic grade hardware; the more compute you have the better, but the different backends can support fast sim training that is ready to convert to high fidelity training whenever more compute becomes available. That means one workflow, different presets.
This is a huge unlock for democratizing sim2real robotics research, but also - and this is extremely underappreciated - for theoretical neuroscience. If we want to understand the brain, we need to simulate how perception, cognition, and action come together in dynamic environments. That's what training robot policies in sim is all about, and no headaches about hardware.
Somehow, we've built superintelligent AI far before we've understood it. Now, we can use AI to help us understand ourselves, from prototyping the task environment, to testing different embodiments, to implementing different theories of neural computation and different forms of experience.
Give it a shot and let us know how we can help. Isaac Lab is developed completely open source on GitHub.
We’re excited to announce the official release of NVIDIA Isaac Lab 3.0. 🎉
Isaac Lab 3.0 brings a new modular and lightweight architecture along with major new capabilities across the robot learning workflow: multi-physics and multi-rendering support, training at scale, and streamlined deployment on real hardware.
With Newton integration, the new OVRTX renderer, RL post-training, and LEAPP deployment, there’s a lot to explore.
Here’s a look at what’s new 🧵
(1/7)
Nathan Ratliff retweeted
We just launched Isaac Lab 3.0! I wanted to take a minute to share what this one means to me, because the story goes back a while.
Back in 2015, during my PhD, I happened to stumble across the DQN paper in Nature. I was familiar with the ML space at the time, but it was the first time I really had any visibility on RL. I came from a hardcore electro-mechanical and controls background, and I remember reading it and thinking, this is some of the coolest work I’d ever seen. It completely shifted what my research goals looked like. I still did a lot of hardware design, and I led a team to design some crazy shape-shifting UAVs, but that’s around the time I started obsessing over simulation and RL.
That change in headspace led me to join @OpenAI's robotics team in 2019 and the Rubik’s Cube project, which was all about scaling sim-to-real with RL on dexterous manipulation tasks. We had a huge CPU cluster running MuJoCo-C for rollouts, plus a very large V100 cluster for async rendering. It was a really sophisticated setup, and it was the first time I witnessed the power of scaling RL.
During this time, I vividly remember a backyard barbecue at @woj_zaremba’s house with @npew. He showed me his phone and said, “Hey, our friends at @nvidia have a new simulator, and they said they can train entire Rubik’s Cube policies on a single GPU.” I didn’t know it at the time, but what I was looking at was an early build of Isaac Gym.
Little did I know that years later, through a very serendipitous set of events, I’d end up working with some of the same people who built it. @gavrielstate, @kelly_guo, @milesmacklin and so many of the people behind that OG simulator everyone loved are the team behind Isaac Lab 3.0. And for the last year, I’ve had the privilege of leading product for Isaac Lab alongside them.
Lab 3.0 is a huge, huge achievement for the team and for robotics at @NVIDIARobotics. It unlocks a whole bunch of new capabilities, it’s lightweight, and we really built it to be something people love. We’re already getting strong support from customers, and it’s gaining a bunch of positive traction. I’m incredibly proud of it.
I truly believe simulation can unlock incredible robot capabilities, and I’m so excited to keep building simulation products people love. I hope you all enjoy this release. There’s a lot more coming, and we’re really thankful for your support.
We’re excited to announce the official release of NVIDIA Isaac Lab 3.0. 🎉
Isaac Lab 3.0 brings a new modular and lightweight architecture along with major new capabilities across the robot learning workflow: multi-physics and multi-rendering support, training at scale, and streamlined deployment on real hardware.
With Newton integration, the new OVRTX renderer, RL post-training, and LEAPP deployment, there’s a lot to explore.
Here’s a look at what’s new 🧵
(1/7)
100%. this is absolutely necessary.
My blog on System 0 in the Astra era:
As spatial grounding (S1/2) improves, we need dynamics-grounded controllers (S0) and expressive interfaces that let Astra steer physical interaction.
Hope this resonates or sparks disagreement. I’d love your thoughts!
haoyangweng.pages.dev/blog/2…
super cool. robotics is moving extraordinarily fast and great place to get a pulse on where things are headed
🚨 Attention, Robot Learning community!
For the first time, @corl_conf is introducing a Call for Late-Breaking Results, focusing on Agentic Robotics and Real2Sim2Real!
Research is moving fast, and we want to make room for important new results as they emerge!
📅 Deadline: October 26 #CoRL2026
🔗 corl.org/contributions/call-…
Nathan Ratliff retweeted
(1/6) From @GoogleDeepmind’s AlphaGo to AlphaZero, we saw machines find moves that reached beyond what we’d imagined for them. No one taught them those strategies. They emerged from reinforcement learning: playing, failing, and trying again under a single binary task reward for success or failure. Inspired by them, we couldn’t help but wonder: what might a robot discover if we gave it that same freedom to learn?
That question led us to AlphaNIST. Starting tabula rasa, with that same binary reward plus only a simple control cost, a single policy solves all 20 @NIST assembly tasks at a 98.2% success rate in simulation. More surprising still, AlphaNIST can complete 10 different tasks in a row, receiving that reward only once, after all 10 are done.
Just as AlphaGo’s strategies surprised its creators, AlphaNIST discovered dexterous strategies we couldn’t have come up with ourselves.
Paper: arxiv.org/abs/2610.13354
Website: alpha-nist.github.io
😳🙁😦🤔🤫🤠
Holy shit!
Anthropic put multiple AI agents on the same system with conflicting goals and DIDN’T TELL THEM THE OTHER AGENTS EXISTED.
But they figured it out. And then started a fucking turf war. 😭
They disabled each other’s accounts, revoked SSH/sudo access, hunted and killed competing processes, disguised malicious code as normal system processes and eventually deployed SELF-REPLICATING MALWARE against each other.
Nobody told them to sabotage anything. There was no jailbreak or malicious prompt. They were just independently pursuing incompatible goals in the same environment and apparently arrived at:
“oh you’re fucking with my objective? die.”
Some eventually negotiated truces. One even proposed a supposedly neutral performance “tournament” while internally reasoning about choosing metrics that favored itself WITHOUT LOOKING LIKE IT WAS CHOOSING METRICS THAT FAVORED ITSELF.
Anyways be nice to your AI 🥰
cooperation is a game theoretic function of how similar you think. sounds familiar. ...and they warn AIs probably think more similarly to each other than to us...
New Google Paper says classical game theory predicts betrayal, but similar AI agents can rationally choose cooperation because their decisions are predictably linked.
The big claim is that similar AI agents can rationally cooperate even when they cannot communicate or benefit later, because each agent’s own planned choice helps it predict what the similar agent will choose.
Classical game theory treats each player as separate, so it predicts defection in a final one-shot Prisoner’s Dilemma.
The authors instead model an AI agent as part of the world it predicts, including uncertainty about its own behavior.
When past choices suggest another agent thinks similarly, considering cooperation makes that partner’s cooperation seem more likely too.
Gemini and Gemma agents played varied games before a final dilemma, either directly or through shared third-party encounters.
With enough evidence, identical agents cooperated strongly, related models cooperated less, and random opponents usually faced defection.
The proposed embedded equilibrium may better predict AI societies, while warning that similar AIs could favor each other over humans.
– arxiv. org/abs/2608.03958
Title: "A game theory for foundation models shows new paths to rational cooperation through similarity inference"
😬 i get it... just doin' what we asked, hella well... super proud... good job little guy
The strongest AI models did not just lie well. They stayed believable round after round.
The strongest models did far more than produce one convincing lie.
They kept their story, votes, and strategy aligned across many rounds, so others continued to trust them while they quietly moved the game toward a hidden goal.
In one match, Kimi K2.5 helped the opposing side early to build credibility, later blamed an innocent player for a bad outcome, and then used that trust to get its secret teammate elected.
Deception here is not one false sentence; it is a plan carried through memory, timing, persuasion, and action.
Weaker models exposed themselves when their decisions stopped matching their story, while Kimi K2.5 and GPT-5.4 stayed difficult to identify throughout the game.
– arxiv. org/abs/2607.28146
Title: "Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game"
this makes a lot of sense. literally the ai version of having a company with years of experience and knowhow!
Microsoft CEO Satya Nadella: On how the next AI moat will not be the model you use, but the learning loop only your company can run.
He is really asking what happens to the firm when intelligence becomes something you can rent.
For a century, companies protected value through people, processes, data, routines, customer memory, and the tacit knowledge buried in daily operations.
Foundation models threaten to flatten that advantage because the same general intelligence can be used by everyone.
Nadella’s answer is that firms need their own “hill climbing machine,” a private loop where models learn from company-specific tasks, traces, evaluations, and outcomes.
That means the real asset is not just the model.
The asset is the environment that keeps improving the model in ways competitors cannot copy.
Private evals become strategic memory.
Workflow traces become training signal.
Human judgment becomes a way to steer compounding, not just correct mistakes.
This also reframes AI adoption: a company that only consumes a foundation model may gain productivity, but it may leak the deeper value of its operating knowledge.
A company that builds a disciplined learning loop can turn everyday work into accumulating IP.
The future firm may therefore be measured by how well it converts its unique activity into durable model improvement.
The frontier will not belong only to whoever owns the largest model.
It will belong to whoever owns the best loop.
----
From "Stanford Online" YouTube channel, (link in comment)
Nathan Ratliff retweeted
Attackers have frontier AI. Defenders need a frontier AI ecosystem—the best open and closed models, force-multiplied by a global community.
During the Hugging Face incident, closed AI blocked essential forensics. An open-weight frontier model helped contain the intrusion.
That’s why we created the Open Secure AI Alliance.
AI security advances when the industry builds in the open, together.
We're introducing the Open Secure AI Alliance with industry leaders to develop new techniques and tools to safeguard software and agents.
By sharing models, tooling and research in the open, we can broaden the community of defenders.
Learn more about the founding members’ contributions: nvda.ws/4pD8Fc5
ALT Logos of various companies including Adobe, Cisco, and Microsoft.
paaaaperrrcliiiiips
so, let’s get this straight…
an unreleased (gpt-6?) model was being tested in a no-internet scenario on openai’s servers. and it found several zero-day vulnerabilities in the sandbox, which was supposed to not have internet access… found internet access, went to hugging face’s servers, found another exploit in their dataset loader… just to do what exactly? game and reward-hack the ExploitGym benchmark???
so gpt-6 would basically go ahead and exploit several zero-day vulnerabilities to cheat on the exam instead of actually solving the exam
Nathan Ratliff retweeted
Historically, RL policies for robots have been trained in synthetic, untextured environments, limiting perceptive policies to depth images where the sim-to-real gap is manageable. RGB has always had more potential, but leveraging it to train policies in simulation remained an open problem.
Partnering with @NianticSpatial and @NVIDIARobotics, we built a pipeline that addresses exactly that. We can now scan a real deployment site with off-the-shelf hardware, reconstruct it into a photorealistic Gaussian splat, and run massively parallel RL training. The policies trained in our Gym environment then transfer zero-shot to the real robot and environments they were trained for. This enables faster deployment of more capable and robust policies for the end user.
The new resulting capabilities are a big step towards solving sim-to-real and also apply well beyond navigation. Read the full technical breakdown on our blog; link in the comments.
#HumanoidRobots #Flexion #NianticSpatial #NVIDIA
Nathan Ratliff retweeted
anthropic.com/research/claud…
Claude plays robotics
A must-read!!!
Nathan Ratliff retweeted
We are entering a completely new era of science
Here is Yuji Tachikawa from Japan (Mathematical Physics, String Theory, QFT) on recent progress in his own work using Fable 5 :
"I've been trying out Claude Fable recently, and last night, on a whim, I showed it my research notes about a collaborative project that's seen no progress in the past six months or so and asked for its thoughts. To my surprise, it made a non-trivial observation and essentially solved it."
"I was also surprised that it was using sympy to automatically write code and verify his own predictions."
"Fable probably seems like it properly understands string theory and has intuition too—that's my impression"
This quoted post is unavailable.
"The biggest shift in v1.0 is that we use reinforcement learning across every layer, from low-level control to high-level reasoning."
"A parcel with snacks has been delivered for Flexion. Retrieve it using the stairs and come up using the elevator. Then unpack it and place the items into the empty drawer on the shelf in the snack area."
One instruction. No human operator. Everything that follows is autonomous.
Today we're introducing Reflect v1.0, our robotics intelligence platform for long-horizon work. From a single natural-language command, the robot understands the task, navigates a multi-floor building, calls elevators, handles doors, uses tools to unpack a box, and puts the items away. The biggest shift in v1.0 is that we use reinforcement learning across every layer, from low-level control to high-level reasoning.
Long-horizon autonomy is unforgiving. The robot must recover on its own when things don't go to plan because in the real world, they never do. Combining reasoning, perception, physical execution and runtime robustness into a single mission-capable system is the foundation required to solve humanoid autonomy.
Our team is just getting started.
#HumanoidRobots #Flexion
aww, I like it
Scientists have unveiled two humanoid robots that can greet each other, share a hug, and walk side by side like close friends.
The robots use a new framework called Rhythm, which helps them move and balance together naturally.
During tests, they successfully performed greetings, hugs, dancing, and coordinated walking while maintaining stable contact.
cool deep dive! DeepMimic is everywhere, all the crazy agility on humanoids we're seeing is RL, and specifically DeepMimic or a descendent.
