Ata retweeted
Today we launch #MemoriesUpload.
Robots need internet-scale first-person video of how people get things done. Collecting this at high quality takes hardware, model and infra working together.
That's what we've spent over a year building. And already, 3 of the top 5 frontier robotics labs are training models with the data.
can confirm its a race to the bottom for raw, esp mono rbg
it seems real world data collection as a business is now becoming increasingly commoditized, there is no differentiation outside of owning a unique data source or relationships, all of which are short lived in monetary value
though, turning data into rl tasks (envs) requires an entirely different profile/skillset to succeed at relative to just buying company data, and the comparative advantage of doing it well is only going up as raw data gets easier to source
it’s not that envs by themselves are a durable business, rather that creating (good) RL tasks worth training frontier models on has genuine engineering/skill challenges to be able to do it at scale for long
there is a reason many of the large data vendors that have grown by predominantly acquiring real-world data and/or sourcing experts are now all becoming "research" orgs (not to mock, a few of them look like they're doing it well)
it's a similar trend as to what's happening in ai infra, where sandbox companies are needing to move up the value stack (e.g. sandboxes -> cloud agents)
the value stack for data probably looks something like:
raw data
de-identification
curated datasets
and then RL tasks which has a whole skill/difficulty stack in of itself (grading, simulating services, fidelity, etc.)
in a sense it's quite meritocratic, if you can make great rl tasks that make models more economically useful in the real world then you will have uncapped demand for it - though this is much easier said than done.
a year from today, the soul of a fast growing data company is likely much more research eng shaped than it is operations shaped, whereas operation skillsets have dominated for the past few years.
the offering to labs from these vendors will genuinely just be “you make $100B from your api platform, buy my tasks for $10m that will make your model 1% more economically useful" and in the eyes of the lab this is an amazing deal (especially in the face of a competitor getting that same taskset)
3
123
Published an 1500+ episode open source ego data set for robotics about a week ago. Zero marketing... Has 2k dls already
You can just do things
huggingface.co/datasets/Cary…
1
101
Tried zero-shotting a VLA in an SO-101 MuJoCo sim. Couldn't find a working one, so we open sourced one instead. a sim, demo collector, training config + eval harness.
0% zero-shot -> 30% success after training
Check it out and feedback welcome!
ataghof.github.io/molmoact2-…
Meet VLA-JEPA, the first world model policy in LeRobot 🌍
Most VLAs just learn observation → action. VLA-JEPA also learns action-relevant dynamics: during training, a JEPA world model has to predict upcoming frames in latent space from the model's own actions. Then, at inference, the world model disappears entirely, leaving a standard, fast VLA: Qwen backbone, action head, nothing extra.
In our demo it was fine-tuned on just 13 examples and ran in real time on an NVIDIA DGX Spark. On a consumer RTX 3080, it hits 10Hz using under 6GB of VRAM.
World-model supervision during training, zero extra cost at inference. That's the whole trick.
3
7
322
Github: github.com/ataghof/molmoact2…
Weights: huggingface.co/ataghof/molmo…
Training Data: huggingface.co/datasets/atag…
1
3
66
We scaled a robot model natively to 8,000 timesteps of context, 5 minutes worth of muscle memory, with constant inference cost. Robot policies used to live their lives a few frames at a time (< 0.1 sec), instantly forgetting what just happened. We pushed to 3 orders of magnitude beyond SOTA.
Introducing RoboTTT. Test-Time Training (“TTT”) carries a tiny model *inside* the model. Every incoming sensor reading triggers one gradient step on that tiny core, so the history keeps getting compressed into its weights. The hidden state has a fixed size (literally a small neural net), so the robot can “grok” arbitrarily long experience with little overhead. Learning continues indefinitely after deployment.
We can then put an entire video in context as prompt! RoboTTT enables one-shot in-context learning from human video: in circuit board assembly, a human demonstrates a never-seen configuration once, and the robot imitates it faithfully.
Humans drop things all the time, but we pick them up so fast that we don’t even notice. That reflex to fix is half of our physical competence. RoboTTT shows self-improvement on the fly: the robot is skilled at recovering from its own errors mid-episode, and each fix enters its context to inform the next move. The TTT core distills a general-purpose, failure-to-correction mapping from the training data.
One more thing. What excites me the most is a new Context Scaling Curve: from 128 to 8K timesteps, closed-loop performance hill-climbs steadily with no sign of saturation. 8K-context pretraining beats 1K by 62%. What LLM enjoys, robotics should too. Soon, even 1M context is not a fantasy.
Deep dive in thread:
🚢🚢🚢
Your robot can now see in depth 👀📏
LeRobot v0.6.0 adds real end-to-end depth support. Plug in an Intel RealSense, set use_depth: true, and depth maps join your RGB observations automatically: captured in millimeters, compressed into compact 12-bit depth video alongside your normal camera streams, and decoded back to physical units at training time.
Depth renders live while you record and in the dataset viewer, and it works across SO-100/101, Koch, OpenArm, reBot, Unitree G1 and more. You also get full control over how everything gets encoded, RGB or depth, codec, quality, presets, all of it.
More on how it works: huggingface.co/docs/lerobot/…
1
102
mans out here soliciting cold emails for data
Should tell you all you need to know about the need for niche datasets
If any lab or company develops RL environments and long-horizon coding task datasets, feel free to reach out to me directly 📮 -> "yuanhe at minimaxi.com"
3
170
Weekend project unlocked
Meet VLA-JEPA, the first world model policy in LeRobot 🌍
Most VLAs just learn observation → action. VLA-JEPA also learns action-relevant dynamics: during training, a JEPA world model has to predict upcoming frames in latent space from the model's own actions. Then, at inference, the world model disappears entirely, leaving a standard, fast VLA: Qwen backbone, action head, nothing extra.
In our demo it was fine-tuned on just 13 examples and ran in real time on an NVIDIA DGX Spark. On a consumer RTX 3080, it hits 10Hz using under 6GB of VRAM.
World-model supervision during training, zero extra cost at inference. That's the whole trick.
70
Ata retweeted
He worked on Roblox’s world model.
I don’t usually share jobs moves, but this one caught my eye.
Amongst the people collecting the data to make world models they think @1x_tech has the best world model outside of Tesla.
And the cuddliest robot.
I watched @karpathy hug one.
I’m excited to announce that I’ll be building robots as VP Engineering at @1x_tech. Rapid progress in world models will create a robotics inflection point.
Very excited to work with @BerntBornich and the stellar 1X team to bring humanoid robots to homes very soon. We’re hiring.
Autoresearch for robotics?
Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library.
ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent.
"Trained model" is a repo of sensorimotor skills instead of floating weights.
“Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches.
Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;)
Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours!
Deep dive in thread:
90
This is going to be huge for scaling tactile data collection... looking forward to getting my hands (literally) in these
In Sep 2024, I left a job I loved to start @proceptionAI with one obsession: to give robots hands that actually work.
20 months. 20 people in Mountain View. A lot of Coke Zero.
Today we ship.
1
86
Wonder how much it costs to train models on this huge data set 🤔
1/ Introducing HIW-500 (Humanoids-in-the-Wild 500):
the largest open-source humanoid teleop dataset collected in real homes
Built w/ @UnitreeRobotics @huggingface across 12 homes in Southeast Asia, it covers:
> 500+ hrs
> 23K+ episodes
> 10+ TB
> 10+ household tasks
1
78
Either Meta trains enough AI data center technicians through their AWA program or @hebbyrobotics builds them
The scarce thing in a data center is not manpower, but instinct that only comes from years on the floor.
@kstonekuan and I spent the past month with data center operators and industrial robotics startups. Most robotics companies are focused on robots as a productivity amplifiers: 24/7 uptime, five days of work done in two. Few are focused on the potential of robots to change how people work altogether.
We want to show what it looks like to rethink human-robot collaboration, using AI so a shrinking pool of experts can meet the increasing demands of future infrastructure.
1
93
This is pretty interested, what if like AI becoming a standard part of all businesses custom robots do as well?
Today we're starting something new at Fuse. We're building robots.
First, why this matters, because it's easy to get wrong. Skilled trades are ageing out faster than they're being replaced. The average electrician is over 45, the average plumber is over 50, and every year we lose more of them to retirement than we train. At the same time the world needs far more of them, not fewer, to build for the surge in energy demand ahead of us. You cannot build power plants, grid and AI data centres without them, and there are not enough of them.
So we're attacking this from both ends.
We're building a training centre in Birmingham, the first of many: untrained human in, skilled tradesperson out. We grow the workforce.
And we're building robotics to amplify what our technicians can do. We don't have enough skilled people as it is. The goal is to make them more productive. Our technicians are at the core of Fuse's mission of acheiving low cost energy and energy abundance.
Why we think we'll win where standalone robotics companies have struggled:
1. We already generate the data robots need. Robotics is bottlenecked on real-world physical data, and you cannot scrape it off the internet. Our technicians do real physical work every day across building power plants, electrical work and energy hardware installs. Only a handful of companies have real physical data. Rarer still to have it across this breadth of skilled trades, on sites you own.
2. We can deploy fast, and we learn faster. Standalone labs spend years convincing big customers to let them deploy on site, often against legacy resistance. We own the worksites, so we build, test and deploy on real Fuse jobs from day one. Continuous deployment means an extremely fast learning rate: more deployment, more data, faster improvement.
This is greenfield. Everything is open for the new team to define, from initial use case to policy choice to the embodiment itself, including whether we build in-house or partner. We're open to working with the best robotics and world model companies out there. The team will work directly with me.
Energy is the bottleneck for AI. Our goal is to unleash it, and robots are how we make sure it never becomes one again.
We're hiring the founding team now. If you want to build robots that ship into the real world from week one and make skilled trades more productive, come talk to us.
1
70
Finally giving @PhoenixTrade a shot. Volume is picking up and feels pretty snappy for a mainnet SOL perps product
58













