Leo Dong retweeted
This is rad!
Leo Dong retweeted
Robots making rapid progress in "learning to learn"
Leo Dong retweeted
What can a robot do without vision?
We gave our robot a sense of touch and showed it can recognize and retrieve objects with its cameras blindfolded.
Introducing Rhoda’s new tactile sensor—it feels pressure, texture, vibration, slip, and temperature, just like your fingertips.
If your robot does this, cut it some slack! Humans do it too when our fingers are anesthetized and lose our sense of touch.
That's why we are bringing tactile sensing to our Direct Video-Action model at Rhoda. Read more in our blog post: rhoda.ai/research/giving-rob…
Leo Dong retweeted
Touch is essential to human manipulation. Yet, we have so far not had a tactile sensor that can do what human skin can: sense texture, firmness, vibration at a viable form factor. At Rhoda, we have built a new sensor in-house that can do all of these things, while being much much smaller than commercially available ones!
Leo Dong retweeted
Such a cool demo of tactile sensing: recognizing and retrieving objects by touch alone! Congratulations to the Rhoda team!
Leo Dong retweeted
Exciting to see how naturally DVA extends to new modalities! Most robot policies still learn without touch, even though it matters most when vision falls short: occlusion, contact, etc.
It was so much fun to see this come together, congrats @leooeld @AndrewEJXie @DaveLindell!
When the project first started, I knew nothing about tactile sensing, and this video @DaveLindell shared really opened my eyes to just how crucial touch is to human manipulation. If you anethesize your fingers and take away touch, you can't even pick up everyday objects! youtube.com/watch?v=0LfJ3M3K…
Leo Dong retweeted
We gave our robots a sense of touch!
Excited to finally share this work with @AndrewEJXie, @leooeld, and the @rhoda_ai team!
Some technical details on the tactile sensor and robot policy: 🧵
(4/n) In the DVA paradigm, learning to predict the next tactile frame is thus equivalent to learning a robot policy that uses tactile signals to guide its actions. We test how much the policy actually uses tactile signals (instead of ignoring them) with an extreme challenge: we ask the robot to perform a manipulation task entirely without vision. With its cameras blindfolded, the robot has to feel around inside a box, search for objects, and pick out only the textured balls from a mix of balls.
(5/n) As humans, our sense of touch complements vision and is essential in its own right, allowing us to manipulate objects even when we can’t see them clearly.
Similarly, our robots can combine vision and touch for contact-rich industrial manipulation, where poor lighting or occlusion can impair vision.
Our DVA model can now imagine what the robot will see—and what it will feel. Read more: rhoda.ai/research/giving-rob…
(1/n) Watch our robot search a box and find textured balls, entirely without vision!
Over the past few months, it’s been incredibly fun to work on bringing touch into our Direct Video-Action (DVA) model. Here’s how our video model is “seeing” through touch 🧵
(3/n) The connection to touch is in the representation. Contact measurements form spatial maps that change over time, which we can think of as a “tactile video.” DVA’s web-scale pre-training gives it a foundation for modeling extremely diverse web videos, making it easy to learn to predict tactile videos through post-training.
(2/n) DVA uses a causal video model as a robot policy. It is a next-frame generator that imagines what the immediate future should look like. An inverse dynamics model realizes that imagined future by translating into actions. The robot moves, observes the result, and the video model predicts again.
Extremely rigorous study. We’ve shown that under the DVA paradigm, *web video* pre-training performance is a strong predictor of *real robot performance* in complex industrial tasks after post-training.
When @tongzhou_mu first presented the findings internally, he ended the meeting with: “At Rhoda, we do real science.” 🐐
Does scaling pre-training on general web video improve a complex manipulation task in real deployment?
We scale model size and pre-training compute, and test on one industrial task.
Yes. The better a pre-trained model predicts web video, the better its post-trained policy. 🧵
ALT Seven pre-trained video-model checkpoints on one chart. Task performance on a real robot (at-speed completion rate: every requirement met within 100 s, 0 to 100%) rises with pre-training quality (DINO FD on held-out web video, log scale, axis reversed so better is to the right). XS, S, M, L are model sizes. The four M points are one model at 0.08×, 0.18×, 0.37× and 1× of its pre-training compute.
We got our most senior “summer intern” ever😂. Welcome!
I am thrilled to join rhoda.ai @RhodaAI as an advisor, where I am helping harness the abilities of large-scale pre-training and video models for robotics, putting many of my lab's research learnings of the past few years into practice! I will be in-person at the Mountain View office for part of the summer - reach out if you want to chat :)
