Praneeth Samineni (@praneethsamine) header
Praneeth Samineni (@praneethsamine) profile picture

cs & math @georgiatech ‘30 365 days of robot learning 46/365

Atlanta, GA
Joined January 2022
Day 32/365 of robot learning. I got an so-101 arm to play ping pong! The video was the first successful run, but it’s been averaging around 13 bounces with a high score of 23 I’ll make a write up on this soon, but in the mean time I’ll keep making the policy more robust!
Day 31/365 of robot learning. Today marks one month since I started this challenge! I'm still working on getting my so-101 robot arm to bounce a ping-pong ball using zero real-world demos, purely through RL trained in simulation. It's not fully working yet, but I'm slowly making progress and solving one problem at a time. The arm can now launch the ball about half the time, bounce it up to 3 times in a row, and even managed to balance it on the paddle for 23 seconds once. Most of the problems haven't even been with the RL algorithm itself, but rather getting the simulation to match reality. For example, the real paddle and ball aren't as bouncy as in simulation, and the servos are much slower under load than I modeled. Even getting the paddle perfectly flat was annoying since being off by just a few degrees would make the ball roll off. I've also spent a lot of time debugging camera calibration, joint offsets, and servo dynamics. I even ended up using the ball itself as a level to calibrate the paddle. Safety was another interesting problem, especially since I have to drop the ball onto the paddle by hand. I made the arm wait completely still until the camera detects the ball, rather than immediately executing the policy. I also trained with simulated hand disturbances, moments when the ball is hidden from the camera, and false detections to make the policy more robust. On the real arm, I've added bounded joint movements, smooth transitions, and automatic recovery when the ball is lost. I'm still working on explicitly detecting when my hand has left the workspace before allowing the arm to move. I'm now retraining with a more realistic servo model to hopefully get it working soon. Also, on a side note, I genuinely don't know how people online doing similar challenges manage to balance a normal work or college life with research and consistently working on side projects. It's becoming pretty hard trying to balance research, open-source work, hackathons, classes, the gym, trying to go on a cut, and all the other responsibilities. But looking back, I've grown incomparably more over this past month. So I'm going to keep going for the next 11 months until I finish this challenge. I definitely plan to keep going beyond the 365 days, but I also really need to figure out how to make all of this sustainable. Also thanks to everyone who provided me with resources and guided me along the way!
40
27
12
644
65,468
Day 45/365 of robot learning. Today I finished reading AW-0. One of the things I found pretty cool was the hybrid execution setup. AW-0 can either predict actions directly through its learned WAM policy or execute generated code through a robotics framework with perception and motion-planning tools. At each replanning step, it checks whether the generated code passes validity and progress checks, otherwise it falls back to the WAM. On RoboTwin 2.0, WAM-only achieved 80.8% success in clean environments compared to 63.1% for hybrid, but under randomized conditions, hybrid performed better at 29.2% versus 20.6%. I wonder if this was just because agentic frameworks are better with OOD tasks. I think the two configurations also use different training demonstrations, so it's not a completely controlled comparison. Tomorrow I want to look more into how AW-0's agentic pathway compares with ASPIRE, since both use code-based robot skills but approach the problem pretty differently. Will post any results I find tmw. There's also an interesting interpretability question here around whether the generated code actually provides meaningful planning information to the action policy, or whether the model mainly benefits from learning to predict code during training.
Day 44/365 of robot learning. Today I looked more into interpretability for world action models, and it almost feels like a goldmine with how much there is still left to explore. A lot of what I’ve been reading focuses on whether generating future video helps task success and how much inference time it costs. But there are so many properties of these models that those measurements don’t tell us much about. I mainly spent some time exploring papers like FloMo, Tapestry, and FasterWAM, which take different approaches to using pretrained video models for robot control. Tapestry generates actions without generating video at inference, while FasterWAM explores efficient future conditioning. FloMo predicts 3D scene flow, representing how points in the scene move in 3D, instead of full future frames. With all these different approaches, it makes me want to understand more precisely what information each representation ends up giving to the policy. For example, what information from the video model makes it into the action policy, and which parts does the policy use? How much do object motion, geometry, or visual details influence its actions? Even if information is present in a hidden representation, that doesn’t mean the policy relies on it. And a higher task success rate alone doesn’t explain what changed internally or why one approach generalizes better to unfamiliar settings. I’d like to see more benchmarks focused on understanding these internal representations and their influence on behavior. There are more specific properties I’ve been thinking about, but I’m keeping those ideas private for now. Will share more once I’ve tested them.
2
20
593
Day 44/365 of robot learning. Today I looked more into interpretability for world action models, and it almost feels like a goldmine with how much there is still left to explore. A lot of what I’ve been reading focuses on whether generating future video helps task success and how much inference time it costs. But there are so many properties of these models that those measurements don’t tell us much about. I mainly spent some time exploring papers like FloMo, Tapestry, and FasterWAM, which take different approaches to using pretrained video models for robot control. Tapestry generates actions without generating video at inference, while FasterWAM explores efficient future conditioning. FloMo predicts 3D scene flow, representing how points in the scene move in 3D, instead of full future frames. With all these different approaches, it makes me want to understand more precisely what information each representation ends up giving to the policy. For example, what information from the video model makes it into the action policy, and which parts does the policy use? How much do object motion, geometry, or visual details influence its actions? Even if information is present in a hidden representation, that doesn’t mean the policy relies on it. And a higher task success rate alone doesn’t explain what changed internally or why one approach generalizes better to unfamiliar settings. I’d like to see more benchmarks focused on understanding these internal representations and their influence on behavior. There are more specific properties I’ve been thinking about, but I’m keeping those ideas private for now. Will share more once I’ve tested them.
Day 43/365 of robot learning. Today I started watching MIT’s Flow Matching and Diffusion Models, but most of my time went into C programming for my computer organization class. I got pretty sick today, so it was a slower day. This course has been on my watch list for a while, and I’m glad I finally started working through it. With how often flow matching and diffusion come up in the robotics papers I’m reading, I want to get more comfortable deriving the math and understanding it. As winter break approaches, my goal is to solidify my fundamentals while continuing research in parallel. Reading all these papers exposes so many gaps in my understanding, so I want to keep working to fill them in. I’m also working on ordering parts for my humanoid, so hopefully there will be more hardware updates soon. On the side, I’ve been exploring tactile sensing with my friend @hemil_thakkar07. He’s handling the 3D printing and building, and we’re using a low-cost design with magnets inside a 3D-printed structure. We’ve been abusing the Georgia Tech makerspace for this. More experimentation on that soon. I feel like this is an area I have to explore because touch could help address so many deployment bottlenecks like plugging cables into server racks or adjusting a part that gets stuck during assembly.
3
1
23
1,547
Day 43/365 of robot learning. Today I started watching MIT’s Flow Matching and Diffusion Models, but most of my time went into C programming for my computer organization class. I got pretty sick today, so it was a slower day. This course has been on my watch list for a while, and I’m glad I finally started working through it. With how often flow matching and diffusion come up in the robotics papers I’m reading, I want to get more comfortable deriving the math and understanding it. As winter break approaches, my goal is to solidify my fundamentals while continuing research in parallel. Reading all these papers exposes so many gaps in my understanding, so I want to keep working to fill them in. I’m also working on ordering parts for my humanoid, so hopefully there will be more hardware updates soon. On the side, I’ve been exploring tactile sensing with my friend @hemil_thakkar07. He’s handling the 3D printing and building, and we’re using a low-cost design with magnets inside a 3D-printed structure. We’ve been abusing the Georgia Tech makerspace for this. More experimentation on that soon. I feel like this is an area I have to explore because touch could help address so many deployment bottlenecks like plugging cables into server racks or adjusting a part that gets stuck during assembly.
Day 42/365 of robot learning. Today I continued working through AW-0, focusing on how its three transformer experts generate code, future video, and robot actions together. I worked through the math behind how the code expert fills in hidden tokens using masked diffusion, while the video and action experts use flow matching to turn random noise into continuous predictions. Also how to represent clean data mixed with noise and how to derive the velocity the model learns to predict. I continued my review of the standard transformer architecture from yesterday as well, mainly using active recall to rederive the functions behind how attention passes information between tokens and how the feed-forward networks update each token’s internal representation. On the side, I’ve also been learning how probes work from the ground up for my WAM interpretability research, cuz before this I was blindly hoping Claude Code was doing it properly. Looked over how to keep the policy frozen, record its hidden states, and train a small predictor to test what information they contain. I think I'm going to look over interventions next. I’m still rebuilding these foundations, but it’s helping me get more precise about what I understand and what I need to revisit. Idk yet if deriving all this math is contributing to my intuition, or if just the high level understanding is enough when we have a bunch of agents under us. I feel like my undergrad is the one time I can prioritize the learning so I'll do it either way.
2
3
1
103
4,148
Day 42/365 of robot learning. Today I continued working through AW-0, focusing on how its three transformer experts generate code, future video, and robot actions together. I worked through the math behind how the code expert fills in hidden tokens using masked diffusion, while the video and action experts use flow matching to turn random noise into continuous predictions. Also how to represent clean data mixed with noise and how to derive the velocity the model learns to predict. I continued my review of the standard transformer architecture from yesterday as well, mainly using active recall to rederive the functions behind how attention passes information between tokens and how the feed-forward networks update each token’s internal representation. On the side, I’ve also been learning how probes work from the ground up for my WAM interpretability research, cuz before this I was blindly hoping Claude Code was doing it properly. Looked over how to keep the policy frozen, record its hidden states, and train a small predictor to test what information they contain. I think I'm going to look over interventions next. I’m still rebuilding these foundations, but it’s helping me get more precise about what I understand and what I need to revisit. Idk yet if deriving all this math is contributing to my intuition, or if just the high level understanding is enough when we have a bunch of agents under us. I feel like my undergrad is the one time I can prioritize the learning so I'll do it either way.
Day 41/365 of robot learning. As I’ve been working through AW-0 and DreamZero, especially the mixture-of-transformers architecture in AW-0, it’s been hard to develop an intuition because my understanding of transformers and diffusion models was pretty shallow. Most of it came from Andrej Karpathy and 3Blue1Brown videos I watched more than a year ago, so today I went back and reviewed the basics and started rederiving the math for the different steps using. I just started today, so I’ll be doing the rest over this weekend. As I explore these different architectures, it also makes me question how much different parts of them affect the policy, which I’ve been exploring in my WAM interpretability research. I think I’m going to spend a few days soon exploring probes, K/V swaps, activation patching, ablations, and activation steering to be able to experiment more with these architectures. I'm also finally done with exams, so I'll be ramping up on side projects over the next few days!
1
1
15
3,739
Day 41/365 of robot learning. As I’ve been working through AW-0 and DreamZero, especially the mixture-of-transformers architecture in AW-0, it’s been hard to develop an intuition because my understanding of transformers and diffusion models was pretty shallow. Most of it came from Andrej Karpathy and 3Blue1Brown videos I watched more than a year ago, so today I went back and reviewed the basics and started rederiving the math for the different steps using. I just started today, so I’ll be doing the rest over this weekend. As I explore these different architectures, it also makes me question how much different parts of them affect the policy, which I’ve been exploring in my WAM interpretability research. I think I’m going to spend a few days soon exploring probes, K/V swaps, activation patching, ablations, and activation steering to be able to experiment more with these architectures. I'm also finally done with exams, so I'll be ramping up on side projects over the next few days!
Day 40/365 of robot learning. Today I spent some more time going through AW-0, and since I didn't have much time today, I decided to just focus on understanding how its code expert generates structured skill programs using discrete masked diffusion. Unlike traditional autoregressive LLMs that generate code one token at a time from left to right, AW-0 starts with a sequence of masked tokens and gradually fills them in over multiple denoising steps. At each step, the code expert predicts a probability distribution over possible tokens at each masked position, then reveals the tokens it's most confident about. This process continues until the full skill program is generated. During training, it learns this by randomly masking tokens in existing skill programs and predicting the missing ones using the robot's observations, language instruction, state, and previous program context. The code generation also works alongside the video and action experts. While the code expert uses discrete masked diffusion, the video and action experts use continuous flow matching. All three are denoised jointly, with the code representations providing planning context for the predicted future video and robot actions. This means the model doesn't necessarily have to finish generating a program before it starts using that information to predict actions, unlike traditional agentic workflows where a separate agent generates a plan before passing it to a low-level controller. The paper also found that using structured code programs instead of natural-language subgoals improved performance on 7 of 8 real-world tasks. I found it pretty interesting how integrating structured planning directly into a learned policy can improve semantic reasoning and language following without a planning stage like agentic frameworks. Still curious how well this policy scales to longer-horizon tasks, especially when the robot needs to recover from failures and replan without an agent framework. Will post about it in the future as I keep exploring this approach
1
20
1,362
Day 40/365 of robot learning. Today I spent some more time going through AW-0, and since I didn't have much time today, I decided to just focus on understanding how its code expert generates structured skill programs using discrete masked diffusion. Unlike traditional autoregressive LLMs that generate code one token at a time from left to right, AW-0 starts with a sequence of masked tokens and gradually fills them in over multiple denoising steps. At each step, the code expert predicts a probability distribution over possible tokens at each masked position, then reveals the tokens it's most confident about. This process continues until the full skill program is generated. During training, it learns this by randomly masking tokens in existing skill programs and predicting the missing ones using the robot's observations, language instruction, state, and previous program context. The code generation also works alongside the video and action experts. While the code expert uses discrete masked diffusion, the video and action experts use continuous flow matching. All three are denoised jointly, with the code representations providing planning context for the predicted future video and robot actions. This means the model doesn't necessarily have to finish generating a program before it starts using that information to predict actions, unlike traditional agentic workflows where a separate agent generates a plan before passing it to a low-level controller. The paper also found that using structured code programs instead of natural-language subgoals improved performance on 7 of 8 real-world tasks. I found it pretty interesting how integrating structured planning directly into a learned policy can improve semantic reasoning and language following without a planning stage like agentic frameworks. Still curious how well this policy scales to longer-horizon tasks, especially when the robot needs to recover from failures and replan without an agent framework. Will post about it in the future as I keep exploring this approach
Day 39/365 of robot learning. Today I spent some more time understanding the internals of the mixture-of-transformers architecture in AW-0, which jointly generates code, future video, and robot actions. AW-0 has three transformer experts, each responsible for a different modality. The code expert generates skills using masked diffusion, while the video and action experts use continuous flow matching to predict future video latents and robot action chunks. I had some understanding of MoE architectures before, but it was pretty cool seeing how MoT works here. Rather than routing tokens to different experts like a traditional MoE, each modality has its own transformer, and the experts communicate through masked joint attention at every layer. The attention is bidirectional within each expert, but blockwise causal across experts in the order code → video → action. This allows the code expert to provide planning context for both video and action generation, while the action expert can also use the video representations. The paper also showed that this ordering improved performance compared to other attention configurations. All three modalities are refined together during the same iterative denoising process, with the code plan guiding action and video. I think it's a pretty interesting approach to integrating high-level planning directly into a learned robot policy, rather than keeping an agent and WAM completely separate. I guess one tradeoff is the additional inference cost, although their joint denoising approach is actually faster than autoregressive code generation. A lot of exams this week, so I'll get back to my projects this weekend!
1
1
11
1,288
Day 39/365 of robot learning. Today I spent some more time understanding the internals of the mixture-of-transformers architecture in AW-0, which jointly generates code, future video, and robot actions. AW-0 has three transformer experts, each responsible for a different modality. The code expert generates skills using masked diffusion, while the video and action experts use continuous flow matching to predict future video latents and robot action chunks. I had some understanding of MoE architectures before, but it was pretty cool seeing how MoT works here. Rather than routing tokens to different experts like a traditional MoE, each modality has its own transformer, and the experts communicate through masked joint attention at every layer. The attention is bidirectional within each expert, but blockwise causal across experts in the order code → video → action. This allows the code expert to provide planning context for both video and action generation, while the action expert can also use the video representations. The paper also showed that this ordering improved performance compared to other attention configurations. All three modalities are refined together during the same iterative denoising process, with the code plan guiding action and video. I think it's a pretty interesting approach to integrating high-level planning directly into a learned robot policy, rather than keeping an agent and WAM completely separate. I guess one tradeoff is the additional inference cost, although their joint denoising approach is actually faster than autoregressive code generation. A lot of exams this week, so I'll get back to my projects this weekend!
Day 38/365 of robot learning. I was planning to spend more time exploring agentic frameworks for robot control and continue reading NVIDIA's ASPIRE, but ended up taking a bit of a diversion today and started reading AW-0. What I found pretty genius was how it builds on WAMs that jointly predict future video and robot actions, but extends this idea to code generation as well. AW-0 uses a mixture-of-transformers architecture with three coupled experts for code, video, and actions. Rather than having a separate agent generate a plan and pass it to a low-level policy, the code expert generates structured skill programs alongside the video and action predictions. These programs help guide both the predicted future and the actual actions, improving the model's performance. What's even better is that the generated code can also be reused for executable skills in an agentic framework. They also tested a hybrid execution approach where the robot can either execute the generated program or follow the predicted action chunk, depending on whether the program passes certain checks. I didn't get too far through the paper today, but this got me thinking about how much further you could take this idea of jointly predicting other things. What if you could introduce additional experts for things like depth, tactile feedback, or force prediction, all within the same architecture? Of course, I imagine there's a tradeoff between jointly predicting more things and inference latency. AW-0 also introduces some additional latency, with the paper reporting about 40ms of overhead from its code expert compared to its video-action baseline on an RTX 5090. I wonder whether selectively activating different experts depending on the task could make this kind of architecture more scalable. Definitely something I want to explore more! I'm also finally back at my dorm, so I can continue messing around with my so-101
2
1
15
1,269
Yup the architecture wars will never end
Brute force computation is not enough for intelligence. Clever new methods are needed. The science of AI is far from finished. IMHO
1
342
Day 38/365 of robot learning. I was planning to spend more time exploring agentic frameworks for robot control and continue reading NVIDIA's ASPIRE, but ended up taking a bit of a diversion today and started reading AW-0. What I found pretty genius was how it builds on WAMs that jointly predict future video and robot actions, but extends this idea to code generation as well. AW-0 uses a mixture-of-transformers architecture with three coupled experts for code, video, and actions. Rather than having a separate agent generate a plan and pass it to a low-level policy, the code expert generates structured skill programs alongside the video and action predictions. These programs help guide both the predicted future and the actual actions, improving the model's performance. What's even better is that the generated code can also be reused for executable skills in an agentic framework. They also tested a hybrid execution approach where the robot can either execute the generated program or follow the predicted action chunk, depending on whether the program passes certain checks. I didn't get too far through the paper today, but this got me thinking about how much further you could take this idea of jointly predicting other things. What if you could introduce additional experts for things like depth, tactile feedback, or force prediction, all within the same architecture? Of course, I imagine there's a tradeoff between jointly predicting more things and inference latency. AW-0 also introduces some additional latency, with the paper reporting about 40ms of overhead from its code expert compared to its video-action baseline on an RTX 5090. I wonder whether selectively activating different experts depending on the task could make this kind of architecture more scalable. Definitely something I want to explore more! I'm also finally back at my dorm, so I can continue messing around with my so-101
Day 37/365 of robot learning. Today I spent some more time going through NVIDIA's ASPIRE framework, and I'm starting to understand why it was so computationally expensive. For context, since ASPIRE essentially approaches robot learning as a software engineering problem, it needs to use coding agents to generate Python programs that call existing perception, grasping, motion planning, and control tools. The framework has a coordinator that assigns tasks to actor agents, which generate programs, execute them, analyze failures, and debug their code. The robot execution engine also provides multimodal traces from things like depth estimation, SAM3 segmentation, grasp prediction, and motion planning, allowing agents to identify where things went wrong. To keep improving on new and out-of-distribution tasks, the framework also maintains a library of reusable skills and uses evolutionary search to evaluate different candidate programs. I guess the interesting part here is how it continually improves by rewriting its programs based on previous failures, while also building up a library of skills that can be reused for future tasks. With all of these different factors, it makes a lot more sense why it uses so many tokens. 😂 In just one experiment with Claude Opus 4.6, the framework required over 26,000 LLM calls and 21,000 simulator rollouts across 60 tasks. I'm still working through the paper, but I'm becoming more interested in whether these ideas could be combined with low-level WAMs/VLAs, and whether token cost could be reduced. I also saw a recent paper on training transformers on code alongside images, text, actions, etc., so after I finish this I'll check that out.
3
2
2
27
2,078
Day 37/365 of robot learning. Today I spent some more time going through NVIDIA's ASPIRE framework, and I'm starting to understand why it was so computationally expensive. For context, since ASPIRE essentially approaches robot learning as a software engineering problem, it needs to use coding agents to generate Python programs that call existing perception, grasping, motion planning, and control tools. The framework has a coordinator that assigns tasks to actor agents, which generate programs, execute them, analyze failures, and debug their code. The robot execution engine also provides multimodal traces from things like depth estimation, SAM3 segmentation, grasp prediction, and motion planning, allowing agents to identify where things went wrong. To keep improving on new and out-of-distribution tasks, the framework also maintains a library of reusable skills and uses evolutionary search to evaluate different candidate programs. I guess the interesting part here is how it continually improves by rewriting its programs based on previous failures, while also building up a library of skills that can be reused for future tasks. With all of these different factors, it makes a lot more sense why it uses so many tokens. 😂 In just one experiment with Claude Opus 4.6, the framework required over 26,000 LLM calls and 21,000 simulator rollouts across 60 tasks. I'm still working through the paper, but I'm becoming more interested in whether these ideas could be combined with low-level WAMs/VLAs, and whether token cost could be reduced. I also saw a recent paper on training transformers on code alongside images, text, actions, etc., so after I finish this I'll check that out.
Day 36/365 of robot learning. I've had this idea of building a framework combining high-level agentic reasoning with a low-level policy for robot control sitting on the backburner for a while, so I decided to spend today exploring this direction. The first paper I really wanted to check out was NVIDIA's ASPIRE, which uses an agent to generate robot control programs, execute them, analyze failures, and iteratively improve them. It also builds a library of skills and achieves much higher success rates on long-horizon tasks compared to previous methods. The funny part is that after spending so much time researching different robot foundation models, it's a little annoying to see an agentic approach perform so well. Of course, these frameworks are still extremely computationally expensive, and it's not exactly a fair comparison. It also got me thinking about the Bitter Lesson. It feels like everyone who's tried to go against it eventually gets humbled, so it's definitely worth exploring agentic control alongside end-to-end learned policies rather than betting everything on one approach. There's also the systems side I'm interested in. Before robotics, I was really into ML systems and inference optimization, but robotics hasn't really converged on a dominant architecture yet. I think figuring out the right architecture could unlock even bigger speedups than optimizing the systems around existing ones. So I'll hold off on that for now. Anyways, since I'm still in school, I feel like it's the perfect time to explore different research directions and see what works. Still planning to have plenty of fun experimenting with all these models! Today was just a lot of thinking, talking to different researchers and friends about these ideas, but I'll be back on the grind soon. Also, I'm still planning to get my SO-101 to build a sandwich with Dream-1! I came home for the weekend, so I don't have access to my robot arms right now, but I'll get back to it soon.
2
1
1
33
2,325
This is concerning
Video tampering can mislead every frontier model we tested into harmful actions, including a 100% attempt rate on stabbing a baby doll, producing toxic gas, and pouring water into a toaster. We're presenting at CoRL Late Breaking Results next week. 🧵 You Can't See Everything: Perceptual Attacks on Embodied AI
1
1
12
722
Day 36/365 of robot learning. I've had this idea of building a framework combining high-level agentic reasoning with a low-level policy for robot control sitting on the backburner for a while, so I decided to spend today exploring this direction. The first paper I really wanted to check out was NVIDIA's ASPIRE, which uses an agent to generate robot control programs, execute them, analyze failures, and iteratively improve them. It also builds a library of skills and achieves much higher success rates on long-horizon tasks compared to previous methods. The funny part is that after spending so much time researching different robot foundation models, it's a little annoying to see an agentic approach perform so well. Of course, these frameworks are still extremely computationally expensive, and it's not exactly a fair comparison. It also got me thinking about the Bitter Lesson. It feels like everyone who's tried to go against it eventually gets humbled, so it's definitely worth exploring agentic control alongside end-to-end learned policies rather than betting everything on one approach. There's also the systems side I'm interested in. Before robotics, I was really into ML systems and inference optimization, but robotics hasn't really converged on a dominant architecture yet. I think figuring out the right architecture could unlock even bigger speedups than optimizing the systems around existing ones. So I'll hold off on that for now. Anyways, since I'm still in school, I feel like it's the perfect time to explore different research directions and see what works. Still planning to have plenty of fun experimenting with all these models! Today was just a lot of thinking, talking to different researchers and friends about these ideas, but I'll be back on the grind soon. Also, I'm still planning to get my SO-101 to build a sandwich with Dream-1! I came home for the weekend, so I don't have access to my robot arms right now, but I'll get back to it soon.
Day 35/365 of robot learning. I finally got @dreamlabs_ai's Dream-1 api working on my single SO-101 arm, and the next step is trying to make it build a sandwich! I also got the full agentic workflow working. Deepgram handles voice input, my Claude Code agent breaks down instructions into smaller tasks, and Dream-1 executes the robot actions. I'll be going out to buy ingredients tomorrow to try making a sandwich, so hopefully I'll have a video out soon! I also ran into a lot of issues during setup. My Claude Code agent couldn't figure out how to get Dream-1 working out of the box with the mcp (I think it was partly my agent's fault as described in point 1), and it took a lot of debugging. Here are a few things I had to fix, so hopefully this saves everyone else some time: 1) Use Dream's official setup instead of letting your agent build a custom runner. I think my agent built its own because it was relying on older information from when Dream didn't support single SO-101 arms. 2) Double-check joint calibration. 3) Try to match the camera setup to Dream's real robot station as closely as possible. 4) Check IK joint limits and servo settings to avoid bad poses and reduce lag. The last part is especially important because my arm actually folded back into itself due to incorrect IK limits, and it could've easily damaged itself. Hopefully these fixes help anyone else trying to get Dream-1 running on their own arm!
4
1
1
43
2,430
Day 35/365 of robot learning. I finally got @dreamlabs_ai's Dream-1 api working on my single SO-101 arm, and the next step is trying to make it build a sandwich! I also got the full agentic workflow working. Deepgram handles voice input, my Claude Code agent breaks down instructions into smaller tasks, and Dream-1 executes the robot actions. I'll be going out to buy ingredients tomorrow to try making a sandwich, so hopefully I'll have a video out soon! I also ran into a lot of issues during setup. My Claude Code agent couldn't figure out how to get Dream-1 working out of the box with the mcp (I think it was partly my agent's fault as described in point 1), and it took a lot of debugging. Here are a few things I had to fix, so hopefully this saves everyone else some time: 1) Use Dream's official setup instead of letting your agent build a custom runner. I think my agent built its own because it was relying on older information from when Dream didn't support single SO-101 arms. 2) Double-check joint calibration. 3) Try to match the camera setup to Dream's real robot station as closely as possible. 4) Check IK joint limits and servo settings to avoid bad poses and reduce lag. The last part is especially important because my arm actually folded back into itself due to incorrect IK limits, and it could've easily damaged itself. Hopefully these fixes help anyone else trying to get Dream-1 running on their own arm!
Day 34/365 of robot learning. Today I tried getting @dreamlabs_ai's Dream-1 API working again after they released a new setup for single SO-101 arms. I had a cool idea of being able to tell my robot to make me a sandwich! I'm building a small agentic workflow where Deepgram transcribes my voice, then Claude decomposes high-level instructions like "make me a sandwich" into simpler tasks like picking up bread, placing ingredients, and stacking everything together. Each step gets passed to Dream-1, which generates low-level robot actions, while the agent uses camera observations to verify progress and decide what to do next. If the zero-shot policy doesn't work, I'll go ahead and collect demos to fine-tune a policy to execute each step. This also aligns closely with my broader interest in combining high-level agentic reasoning with low-level world-action models (WAMs) for robot control. The Dream-1 integration isn't fully working yet, though. I'm debugging issues with calibration, coordinate frames, and IK. I'm still not sure if the issue is on my end, but I'll share updates as I work through it over the next few days. Meanwhile, my ping-pong RL policy has reached 28 consecutive bounces! I'm messing around more with training by adding additional tracking and aiming rewards, and expanding the training distribution with more varied ball trajectories, randomized bounce physics, and simulated camera noise to improve sim to real transfer.
14
4
3
85
5,402
Day 34/365 of robot learning. Today I tried getting @dreamlabs_ai's Dream-1 API working again after they released a new setup for single SO-101 arms. I had a cool idea of being able to tell my robot to make me a sandwich! I'm building a small agentic workflow where Deepgram transcribes my voice, then Claude decomposes high-level instructions like "make me a sandwich" into simpler tasks like picking up bread, placing ingredients, and stacking everything together. Each step gets passed to Dream-1, which generates low-level robot actions, while the agent uses camera observations to verify progress and decide what to do next. If the zero-shot policy doesn't work, I'll go ahead and collect demos to fine-tune a policy to execute each step. This also aligns closely with my broader interest in combining high-level agentic reasoning with low-level world-action models (WAMs) for robot control. The Dream-1 integration isn't fully working yet, though. I'm debugging issues with calibration, coordinate frames, and IK. I'm still not sure if the issue is on my end, but I'll share updates as I work through it over the next few days. Meanwhile, my ping-pong RL policy has reached 28 consecutive bounces! I'm messing around more with training by adding additional tracking and aiming rewards, and expanding the training distribution with more varied ball trajectories, randomized bounce physics, and simulated camera noise to improve sim to real transfer.
Day 33/365 of robot learning. Today I spent some time reviewing Dreamzero's architecture and took a break from working on projects to focus more on learning and developing a better intuition for the math behind these architectures. For context, Dreamzero builds on pretrained video diffusion models and learns to jointly predict future video and robot actions. Here's the architecture in a simple equation: π₀(o, a | history, c, q) = π₀(o | history, c, q) · π₀(a | history, future video, q) Here, o represents visual observations, a represents actions, c is the language instruction, and q is the proprioceptive state. Essentially, the joint policy can be decomposed into autoregressive video prediction and an inverse dynamics model (IDM). Unlike generating both separately through a video model and an IDM, DreamZero models both through a shared DiT. I mainly reviewed the action encoder/decoder and the joint flow-matching objective. For each chunk k, the paper derives noisy video latents and normalized actions through linear interpolation: zᵏₜ = tₖzᵏ₁ + (1−tₖ)zᵏ₀ aᵏₜ = tₖaᵏ₁ + (1−tₖ)aᵏ₀ Here, the subscript 0 represents Gaussian noise, 1 represents clean data, and both modalities share the same denoising timestep tₖ. We also get the target joint velocity by differentiating with respect to tₖ: vᵏ = [zᵏ₁, aᵏ₁] − [zᵏ₀, aᵏ₀] The model learns uθ to predict this velocity by minimizing a weighted squared-error loss, conditioned on language, proprioception, and clean previous chunks. After spending a lot of time on projects, I plan to spend the next few days going through more of the Dreamzero algorithms and reading other papers on the side to focus more on learning.
7
2
1
57
4,950
Day 33/365 of robot learning. Today I spent some time reviewing Dreamzero's architecture and took a break from working on projects to focus more on learning and developing a better intuition for the math behind these architectures. For context, Dreamzero builds on pretrained video diffusion models and learns to jointly predict future video and robot actions. Here's the architecture in a simple equation: π₀(o, a | history, c, q) = π₀(o | history, c, q) · π₀(a | history, future video, q) Here, o represents visual observations, a represents actions, c is the language instruction, and q is the proprioceptive state. Essentially, the joint policy can be decomposed into autoregressive video prediction and an inverse dynamics model (IDM). Unlike generating both separately through a video model and an IDM, DreamZero models both through a shared DiT. I mainly reviewed the action encoder/decoder and the joint flow-matching objective. For each chunk k, the paper derives noisy video latents and normalized actions through linear interpolation: zᵏₜ = tₖzᵏ₁ + (1−tₖ)zᵏ₀ aᵏₜ = tₖaᵏ₁ + (1−tₖ)aᵏ₀ Here, the subscript 0 represents Gaussian noise, 1 represents clean data, and both modalities share the same denoising timestep tₖ. We also get the target joint velocity by differentiating with respect to tₖ: vᵏ = [zᵏ₁, aᵏ₁] − [zᵏ₀, aᵏ₀] The model learns uθ to predict this velocity by minimizing a weighted squared-error loss, conditioned on language, proprioception, and clean previous chunks. After spending a lot of time on projects, I plan to spend the next few days going through more of the Dreamzero algorithms and reading other papers on the side to focus more on learning.
Day 32/365 of robot learning. I got an so-101 arm to play ping pong! The video was the first successful run, but it’s been averaging around 13 bounces with a high score of 23 I’ll make a write up on this soon, but in the mean time I’ll keep making the policy more robust!
2
1
28
3,130
Thanks for the arm @robo_nine !
Our next SO-ARM102 giveaway recipient Praneeth Samineni: @praneethsamine Excited to support his robotics projects and see what he builds next.
1
40
2,066
Yea we might be screwed
Robotics is solved when world GDP has grown by a hundred orders of magnitude and we've collonized the galaxy, not when SOTA on some benchmark goes from 42.2% to 63.1%
1
6
1,708
Day 31/365 of robot learning. Today marks one month since I started this challenge! I'm still working on getting my so-101 robot arm to bounce a ping-pong ball using zero real-world demos, purely through RL trained in simulation. It's not fully working yet, but I'm slowly making progress and solving one problem at a time. The arm can now launch the ball about half the time, bounce it up to 3 times in a row, and even managed to balance it on the paddle for 23 seconds once. Most of the problems haven't even been with the RL algorithm itself, but rather getting the simulation to match reality. For example, the real paddle and ball aren't as bouncy as in simulation, and the servos are much slower under load than I modeled. Even getting the paddle perfectly flat was annoying since being off by just a few degrees would make the ball roll off. I've also spent a lot of time debugging camera calibration, joint offsets, and servo dynamics. I even ended up using the ball itself as a level to calibrate the paddle. Safety was another interesting problem, especially since I have to drop the ball onto the paddle by hand. I made the arm wait completely still until the camera detects the ball, rather than immediately executing the policy. I also trained with simulated hand disturbances, moments when the ball is hidden from the camera, and false detections to make the policy more robust. On the real arm, I've added bounded joint movements, smooth transitions, and automatic recovery when the ball is lost. I'm still working on explicitly detecting when my hand has left the workspace before allowing the arm to move. I'm now retraining with a more realistic servo model to hopefully get it working soon. Also, on a side note, I genuinely don't know how people online doing similar challenges manage to balance a normal work or college life with research and consistently working on side projects. It's becoming pretty hard trying to balance research, open-source work, hackathons, classes, the gym, trying to go on a cut, and all the other responsibilities. But looking back, I've grown incomparably more over this past month. So I'm going to keep going for the next 11 months until I finish this challenge. I definitely plan to keep going beyond the 365 days, but I also really need to figure out how to make all of this sustainable. Also thanks to everyone who provided me with resources and guided me along the way!
Day 30/365 of robot learning. What do you guys rate the Halloween fit? I couldn’t get my robot arm to bounce a ping pong ball yet. The main issues were getting the initial bounce going and realizing my actual paddle isn’t nearly as bouncy as the one in sim. I’m gonna try training it on some more realistic flicking motions to get the ball bouncing and see if I can make it work!
17
5
2
169
27,742
Day 30/365 of robot learning. What do you guys rate the Halloween fit? I couldn’t get my robot arm to bounce a ping pong ball yet. The main issues were getting the initial bounce going and realizing my actual paddle isn’t nearly as bouncy as the one in sim. I’m gonna try training it on some more realistic flicking motions to get the ball bouncing and see if I can make it work!
Day 29/365 of robot learning. I'm trying to get my robot arm to bounce a ping pong ball using zero real-world demos, purely through sim-to-real transfer. Hoping to get it working this weekend if it's even possible! Got the idea from @sigrobotics, so shoutout to them!
7
3
2
28
8,645