AI Foundations @Waymo CS PhD @UofT '25, @harvardphysics/@harvardmath '18
Mountain View, CA
Joined November 2018
- Tweets289
- Following565
- Followers853
- Likes2.6K
Pinned Tweet
"Trajeglish: Learning the Language of Driving Scenarios" w/ @xbpeng4 @FidlerSanja
Discrete sequence modeling for controlling interactive agents in self-driving simulation
@nvidia @VectorInst @UofTCompSci @SFU
research.nvidia.com/labs/tor…
arxiv.org/abs/2312.04535
1/6
Jonah Philion retweeted
Robotics is entering a transformative period. 🤖
Principles of Robot Autonomy is the culmination of nearly a decade of building a robotics curriculum at Stanford, and we’re thrilled to share this 360-degree program with students, researchers, and practitioners around the world!
Ever wondered what it’s like to learn robot autonomy @Stanford? Now you can.
Today, we’re opening up the robot autonomy teaching materials that my collaborators and I have been developing over the past several years—for anyone, anywhere, completely free.
At the heart of this effort is Principles of Robot Autonomy (PoRA), a textbook inspired by the Stanford robot autonomy course series I’ve been teaching since 2017.
Robot autonomy has evolved from a research discipline into a technology deployed in the real world. Building capable autonomous systems requires more than understanding individual algorithms: it requires understanding how perception, state estimation, planning, control, and learning work together in a complete autonomy stack.
That’s what we set out to teach with PoRA: bringing classical robotics and modern AI together in one coherent framework, grounded in both theory and practice.
Coming soon in print from Cambridge University Press, Principles of Robot Autonomy is also freely available online and comes with a comprehensive collection of open educational resources:
📖 Textbook, arxiv.org/abs/2608.03496 — Foundations of robot autonomy, from classical methods to modern AI-based approaches.
🎥 Video lectures, porabook.com/lectures/ Lectures from my Stanford courses on robot autonomy and on optimal and learning-based control.
💻 Interactive exercises, porabook.com/exercises/ — Hands-on Python implementations and Jupyter notebooks.
📝 Teaching materials, porabook.com/instructors/ — Lecture slides and resources for instructors.
🌐 Explore all PoRA resources, porabook.com/ — Including Ask the Book, an AI study assistant integrated throughout the textbook to answer questions, explain derivations, and connect ideas across chapters.
📣 Read the Stanford announcement: news.stanford.edu/stories/20…
Our goal is simple: to make high-quality education in robot autonomy accessible to students, researchers, engineers, educators, and anyone curious about how autonomous robotic systems work.
I believe that sharing knowledge is fundamental to the mission of a university. Autonomous robotics—and Physical AI more broadly—are among the defining technological frontiers of our time. We want to help more people understand these technologies, build with them, and advance the field.
A huge thank you to my co-authors, @DanieleGammelli, Joseph Lorenzetti, @katielulula, and @GioeleZardini, and to everyone who helped bring this project to life.
Explore the resources, use them in your own learning or teaching, and please share them with anyone who might find them useful!
#Robotics #PhysicalAI #ArtificialIntelligence #AutonomousSystems #Education
@Stanford @StanfordEng @StanfordAILab @StanfordHAI
Jonah Philion retweeted
At @VeedaAI , we’re building for a future where robots learn by interacting with the world, at scale -- in simulated reality.
Getting there means bringing together the brightest minds to tackle some of the hardest problems in Physical AI.
Come build what’s next with us: jobs.ashbyhq.com/veeda-ai
Jonah Philion retweeted
In our Nature paper, we introduce the first superhuman Stratego AI, which we built using general techniques that we developed for RL & test-time compute under imperfect information. 1/N
Jonah Philion retweeted
Today, I’m incredibly excited to introduce Veeda AI, which I co-founded with @FidlerSanja and @ZGojcic .
Throughout my career, I’ve been fascinated by one question: how can visual generative models impact the physical world?
My journey has taken me from images to video to world foundation models.
Now, we’re building toward the next frontier: turning these generated worlds into virtual interactive playgrounds where physical intelligence can perceive, reason, act, and ultimately learn within them at scale.
That’s our ambition at Veeda and I’m incredibly excited to build this alongside Sanja, Zan, and our amazing team.
We’re hiring. Come build it with us.
I’m incredibly excited to introduce @VeedaAI , which I co-founded with my incredible longtime collaborators @ZGojcic and @HuanLing6 .
We started Veeda because we believe robotics will reshape the world—changing how we move people and goods, how we manufacture and build, and how we operate in the physical world.
We also firmly believe that the scaling moment for Physical AI will come from robots learning through interaction with the world.
Just like humans learn from interaction with their environments by trying, failing and trying again, until we succeed. And just like the capabilities of LLMs were truly unlocked when they started training in interactive environments and not only on vast amounts of internet text.
For Physical AI, the central challenge is making this kind of interactive learning possible at scale. The real world is simply not a practical training ground for robots to learn through trial and error. It is unsafe, expensive, and time-consuming. To scale interactive learning, robots will need to learn in simulated reality.
At Veeda, our sole mission is to build simulated reality for Physical AI. Our conviction is that this will become the critical infrastructure layer for all areas of robotics.
As envisioned in pop culture, this means building “the Matrix” for Physical AI, where, through a virtual embodiment, robot intelligence can interact with an entirely virtual world in a million possible ways, learning from its own mistakes.
We believe that World Models are the foundational technology that can make this possible. These generative models learn from massive amounts of sensor and physical-world data to reach the quality, diversity, and physical realism of the simulated reality that robots require.
I could not be more excited to take on this challenge and build this transformative technology alongside an extraordinary team of engineers and researchers who have spent years working on some of the hardest problems across simulation, generative AI, 3D, robotics, and embodied intelligence.
Together with our investors at @khoslaventures and @radicalvcfund , we intend to push the frontier of this technology and build the foundational infrastructure for interactive robotics learning and evaluation at scale.
Come build this future with us.
And to roboticists, help us understand what you need, we’re building this for you.
veeda.ai
Jonah Philion retweeted
The era of Physical AI is coming, and I firmly believe that embodied intelligence will not emerge from imitation learning alone. Just as humans first learn by observing the world and then build deeper understanding through interaction, the breakthrough moment for Physical AI will come through interactive learning.
Real-world trial and error is too slow, expensive, and unsafe. To scale interactive learning, we need to recreate the richness of reality without being constrained by reality itself. We need simulated reality—a place where embodied intelligence can experience situations, make mistakes, learn from them, and try again millions of times before encountering them in the real world.
I am incredibly excited to build @VeedaAI alongside my co-founders, @FidlerSanja and @HuanLing6 , and an extraordinary group of researchers and engineers. Together, we are developing the next generation of interactive foundation world models for Physical AI.
There is no team I would rather take on this challenge alongside, and I couldn’t be more excited for what we’re going to build.
veeda.ai
I’m incredibly excited to introduce @VeedaAI , which I co-founded with my incredible longtime collaborators @ZGojcic and @HuanLing6 .
We started Veeda because we believe robotics will reshape the world—changing how we move people and goods, how we manufacture and build, and how we operate in the physical world.
We also firmly believe that the scaling moment for Physical AI will come from robots learning through interaction with the world.
Just like humans learn from interaction with their environments by trying, failing and trying again, until we succeed. And just like the capabilities of LLMs were truly unlocked when they started training in interactive environments and not only on vast amounts of internet text.
For Physical AI, the central challenge is making this kind of interactive learning possible at scale. The real world is simply not a practical training ground for robots to learn through trial and error. It is unsafe, expensive, and time-consuming. To scale interactive learning, robots will need to learn in simulated reality.
At Veeda, our sole mission is to build simulated reality for Physical AI. Our conviction is that this will become the critical infrastructure layer for all areas of robotics.
As envisioned in pop culture, this means building “the Matrix” for Physical AI, where, through a virtual embodiment, robot intelligence can interact with an entirely virtual world in a million possible ways, learning from its own mistakes.
We believe that World Models are the foundational technology that can make this possible. These generative models learn from massive amounts of sensor and physical-world data to reach the quality, diversity, and physical realism of the simulated reality that robots require.
I could not be more excited to take on this challenge and build this transformative technology alongside an extraordinary team of engineers and researchers who have spent years working on some of the hardest problems across simulation, generative AI, 3D, robotics, and embodied intelligence.
Together with our investors at @khoslaventures and @radicalvcfund , we intend to push the frontier of this technology and build the foundational infrastructure for interactive robotics learning and evaluation at scale.
Come build this future with us.
And to roboticists, help us understand what you need, we’re building this for you.
veeda.ai
Jonah Philion retweeted
I’m incredibly excited to introduce @VeedaAI , which I co-founded with my incredible longtime collaborators @ZGojcic and @HuanLing6 .
We started Veeda because we believe robotics will reshape the world—changing how we move people and goods, how we manufacture and build, and how we operate in the physical world.
We also firmly believe that the scaling moment for Physical AI will come from robots learning through interaction with the world.
Just like humans learn from interaction with their environments by trying, failing and trying again, until we succeed. And just like the capabilities of LLMs were truly unlocked when they started training in interactive environments and not only on vast amounts of internet text.
For Physical AI, the central challenge is making this kind of interactive learning possible at scale. The real world is simply not a practical training ground for robots to learn through trial and error. It is unsafe, expensive, and time-consuming. To scale interactive learning, robots will need to learn in simulated reality.
At Veeda, our sole mission is to build simulated reality for Physical AI. Our conviction is that this will become the critical infrastructure layer for all areas of robotics.
As envisioned in pop culture, this means building “the Matrix” for Physical AI, where, through a virtual embodiment, robot intelligence can interact with an entirely virtual world in a million possible ways, learning from its own mistakes.
We believe that World Models are the foundational technology that can make this possible. These generative models learn from massive amounts of sensor and physical-world data to reach the quality, diversity, and physical realism of the simulated reality that robots require.
I could not be more excited to take on this challenge and build this transformative technology alongside an extraordinary team of engineers and researchers who have spent years working on some of the hardest problems across simulation, generative AI, 3D, robotics, and embodied intelligence.
Together with our investors at @khoslaventures and @radicalvcfund , we intend to push the frontier of this technology and build the foundational infrastructure for interactive robotics learning and evaluation at scale.
Come build this future with us.
And to roboticists, help us understand what you need, we’re building this for you.
veeda.ai
Jonah Philion retweeted
Excited to announce: In 2027 I'll join the UW Allen School (@UWCSE) in Seattle as tenure-track faculty in Computer Science!
My group will advance core geometry processing + 3D AI for visual computing, science, & engineering. I'll be recruiting PhD students in the next cycle!
Jonah Philion retweeted
🚨new position from my students & collaborators! ever since the release of chatgpt in 2022 (wow, 4 years ago!) chatbots are increasingly integrated into people's lives. we call for a shift towards "longitudinal" measurements and mitigations in alignment frameworks 🧵
Jonah Philion retweeted
After almost five years, I recently made the difficult decision to leave NVIDIA.
NuRec grew from an early demo we hacked together in the last minute for Jensen to present at the 2022 GTC keynote into a simulator with real product impact, and later led to InstantNuRec and OmniDreams.
Seeing research move from prototypes and papers into production was incredibly rewarding. I’m grateful to Jensen, @FidlerSanja, @RevLebaredian, Xinzhou Wu and the many others who created an environment that enables ambitious research and allows researchers to share their work through publications and open-source releases.
NVIDIA has built a rare bridge between fundamental open research, large-scale engineering, and real-world impact. It is a truly special place to work, and I’m proud to have contributed to its journey in physical AI and simulation.
Jonah Philion retweeted
Neural networks might speak English, but they think in shapes.
Understanding their rich *neural geometry* is key to understanding how they work – and to debugging and controlling them with precision.
Starting today, we’re releasing a series of posts on this research agenda. 🧵
Jonah Philion retweeted
Data in the cornestone of everything!
We've used NCore across many internal research/product efforts and now it is finally public.
Canonical representation, easy-to-use APIs, random access, streaming, faster than WebDataset and HDF5, pip installable!
github.com/NVIDIA/ncore
A canonical open-source data platform for multi-sensor neural 3D reconstruction is here.
Introducing NVIDIA NCore — a unified data format and APIs for cameras, lidars, radars, poses, calibrations, and labels, built for AV, robotics, and physical AI.
🔗research.nvidia.com/labs/sil…
Jonah Philion retweeted
A new generation in AV simulation is here!
We are announcing AlpaDreams, a real time interactive generative world model for AV simualtion! Just a year ago it took minutes to generate a few seconds of video, today it is real time and interactive!
research.nvidia.com/labs/sil…
Jonah Philion retweeted
A new era for Waymo. We’ve raised $16B to accelerate our mission, valuing the company at $126B. This capital is an investment in a future where more cities get a safer, more reliable way to move. Let's go. 🚀 waymo.com/blog/2026/02/waymo…
Jonah Philion retweeted
With the $16B in new funding, we’re accelerating the deployment of the @Waymo Driver - the world’s most advanced physical-world AI. With nearly 200M fully-autonomous miles under our belt and scaling exponentially, we’re just getting started… trillions of miles and safer roads ahead!
waymo.com/blog/2026/02/waymo…
Jonah Philion retweeted
Information theory often gives unintuitive conclusions when it comes to data. Many of these inconsistencies can be resolved elegantly if we limit the amount of computation the observers can use. Very happy to finally introduce our work on epiplexity! 1/🧵
1/🧵 We are very excited to release our new paper! From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
arxiv.org/abs/2601.03220
with amazing team @ShikaiQiu @yidingjiang @Pavel_Izmailov @zicokolter @andrewgwils
Jonah Philion retweeted
Exciting expansion! @Waymo now serves the whole SF Bay Area Peninsula from SF to San Jose and is taking riders on freeways.
waymo.com/blog/2025/11/takin…
Jonah Philion retweeted
Excited to release new repo: nanochat!
(it's among the most unhinged I've written).
Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.
It weighs ~8,000 lines of imo quite clean code to:
- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.
Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.
My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.
Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.
We'll host 2 interns next summer on @sergioksas's team at Waymo next summer, one of them hosted by myself.
Apply if you'd like to work on problems at the very frontier of ML-based planning for L4 self-driving!
careers.withwaymo.com/jobs/2…
careers.withwaymo.com/jobs/2…
Jonah Philion retweeted
All systems go at @flySFO! We’ve been approved by the airport to begin operations, and will start testing soon. More here: sf.gov/news-mayor-lurie-take…
