curious about ai, robotics and art. design engineer at @heyglif. fun: https://xitter.cf/t.co/mEqjOEj5IJ
Berlin, Germany
Joined November 2013
- Tweets6.3K
- Following1.7K
- Followers3.2K
- Likes39.3K
Pinned Tweet
Astra’s spatial understanding is mindblowing to watch.
three uncalibrated cameras. no intrinsics, no extrinsics.
it nudged the robot arm, learned how each camera sees motion, worked out the brush tip’s 3d offset, and verified it at new angles -- all from one prompt and a tiny follow up.
final result was under 0.2 mm of camera-measured tip movement!
my biggest frustrations with ai tools rn is their opaque memory systems.
chatgpt knows some things about me, claude remembers smth else, claude code has a different system, codex feels like a sprawling web of long-forgotten chats, and codebase docs + skills are yet another system.
i DO want these memories to be contextual and siloed. different models and harnesses have different strengths. i want their memories to reflect that.
but a few things that really bother me:
1. Opacity: how are memories created, curated, updated, and forgotten? i have almost no visibility into this.
2. Context rot: what random memories and old chats are seeping into the current context window?
3. Curation: no tools to curate memories. i'd like to have powerful tools to inspect, edit, prune, and shape these memories.
nil retweeted
kay neurons are interpretable enough that models can modify their own weights toward some desiderata.
blogpost incoming
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
some calibration issues still need to be ironed out, but loving this wet-in-wet watercolor bleed style!
titled "Slight Drift" by Opus
wondering if i should just run a 24/7 agent on my computer to keep checking for and patching vulnerabilities!
having such a bizarre interaction in codex about printing a webpage where Astra and the tab autocomplete model are arguing with each other.
(the previous user messages were also just me tab auto-sending)
100%! especially with complicated creative work -- long videos, games, 3d scenes, -- there’s such a huge variety in output. you can’t really capture all that nuance in a few hundred words of prompts. plus, adding more detail often makes the models overindex on some stray detail.
what actually works much better:
- giving them lots of tasteful references and some high level pointers
- setting up tools so they can see what they’re making
- asking them to regularly critique their own work
- nudging them slightly over time (but not micromanaging)
A big part of eliciting interesting / novel outputs from bit models is what you ~don’t~ prompt. If you curate enough references and give rough directional guidance that is properly pointed, the models can interpolate style in ways you didn’t anticipate and that’s where the magic happens.
So in the case of putting them to work together, the only way to have them reliably use one another is to lose that instruction gap that allows creativity to shine through. In other words, it’s useless to force it for my purposes.
asked Claude to compose music like "Ryuichi Sakamoto's" for this video, and it spent hours going thru his catalog, analysing spectrograms, downloading scores, asking Gemini 100s of questions cuz it can "understand audio", using audio gen models, writing MIDI directly, what not.
i asked Claude Opus to turn this into a Chrome extension.
it basically works on any website. all credit for the original idea, code and motion to @notdetails 🫡
chrome extension (open-source) in 🧵
somehow makes me so happy whenever i see an xArm on the feed!
This fully autonomous demo shows a Bagel Labs foundation model controlling a robot to make coffee.
We previously released WorldDiT, our lightweight architecture for robot foundation models. At the time of release, it was on the Pareto frontier for model size and performance. We have since been refining this architecture to handle longer
horizon, more useful tasks fully autonomously.
Making coffee means getting a whole sequence of actions right. Each step depends on the result of the last, so small mistakes can build up as the task continues. Getting this to work with a small model is especially challenging. This demo is an example of what the model can do today.
If you are deploying robots or have existing hardware you want to make autonomous, we would like to partner with you. bagel.com/partner
as robots get better at dexterity, they'll will use their bodies, actuators, sensors and surroundings in such creative ways that it'd be baffling to our primate brains.
they'll routinely pull out move 37 manoeuvres.
for example: can you fold a tee shirt with just one finger? can you make a meal with just two fingers and one hand? can you write with your eyes closed? they're hard, but not impossible for humans, and in 2 years, it'll be same for robots.
(1/6) From @GoogleDeepmind’s AlphaGo to AlphaZero, we saw machines find moves that reached beyond what we’d imagined for them. No one taught them those strategies. They emerged from reinforcement learning: playing, failing, and trying again under a single binary task reward for success or failure. Inspired by them, we couldn’t help but wonder: what might a robot discover if we gave it that same freedom to learn?
That question led us to AlphaNIST. Starting tabula rasa, with that same binary reward plus only a simple control cost, a single policy solves all 20 @NIST assembly tasks at a 98.2% success rate in simulation. More surprising still, AlphaNIST can complete 10 different tasks in a row, receiving that reward only once, after all 10 are done.
Just as AlphaGo’s strategies surprised its creators, AlphaNIST discovered dexterous strategies we couldn’t have come up with ourselves.
Paper: arxiv.org/abs/2610.13354
Website: alpha-nist.github.io
i've been trying to teach watercolor to the Robot.
we have settled on a new workflow where i just put some colors, sponge, water and paper -- and Opus/Astra run full experiments with dip variants, brush tilt, speed, blend ration, stroke paths, and whatnot.
it is so exciting to see the quality of paintings slowly improve over time