
AI controllability @GoogleDeepMind | ex The M Machine | opinions my own
New Jersey, USA
Joined October 2014
- Tweets7K
- Following1.6K
- Followers6.9K
- Likes65K
Pinned Tweet
I wanted to share something I built over the last few weeks: isometric.nyc is a massive isometric pixel art map of NYC, built with nano banana and coding agents.
I didn't write a single line of code.
Andy Coenen retweeted
I am pretty concerned we are in the early stages of a global cyber security incident and we just don't fully realize the extent of it yet.
Andy Coenen retweeted
Absolutely insane how much leverage there is right now in finding precisely the right abstractions and APIs for things
Andy Coenen retweeted
they told us in math class in the early 2000s that we wouldn’t have a calculator with us always and 20 years later my friends argue whether or not my pocket phone contains a conscious machine god
hell yeah this is awesome… or should i say brilliant, cracking, a true labour of love
Introducing tiny.london
A massive isometric pixel map of London, based on amazing NYC demo by @_coenen from 9 months ago
My agent created the dataset, finetuned qwen image LoRA adapter, did a multiday inference run and built an efficient tile viewer!
the best advice in the age of AI: "dream bigger"
ok so... there was a 2 day hackathon at work last week...
i built an infinite, 3D, ai native mmo called nocturne... it's set in the world of inkwell.
the server capacity is... pretty good..? i tested it with 1000 connections in the cloud.
it's written in a programming language i wrote (called ink).
it uses polygons & gaussian splat *frostings* for the graphics.
it has an adaptive music system that changes intensity and mood based on the gameplay.
the combat is kind of realtime dnd-like?
all of the npcs are powered by llms, you can talk to them, trade, fight them... they remember you..?
they have little neural nets inside them and they optimise for survival and happiness.
there's *trillions* of combinations of characters... *hundreds of millions* of spell combinations.
all of the creatures have a genome, they reproduce and have mutations creating new creature types...
you can catch them like in pokemon... there's semantic crafting and alchemy... you can be turned into a vampire and burn in the sunlight..?
there's a day & night cycle... seasons... fires spread, rain puts them out, ice freezes over lakes...
you can become a werewolf and infect others? or a weredragon? even a wererat.
you can ride on goats, sheep, horses, eagles and dragons...
aliens can abduct you or the farm animals...
and when you do... you can discover and explore their home planets with different physics.
there's giant colossal beasts to fight, demigods with trials that give you world changing powers...
you can get portals in a bag that take you to home's you can customise like in tiny glade...
oh and there's an api... so people can connect their local agents to it and build their own worlds...
rift's open up and invasions happen. villages learn traditions and will fear you if you treat them badly...
there's open pvp everywhere and you drop all of your gear if you die.
if you try to do pvp in the main cities, then mages will spawn in and try to kill you.
oh... there's also semantic scrying.. you can get any item and cast a spell and it opens a portal to the closest thing that is semantically similar... got a bone? maybe it'll take you to a graveyard... a log? a forest, etc.
i feel like i haven't even *scratched the surface* with how deep this game is, but this post is pretty long... so... *dream bigger*.
Andy Coenen retweeted
Multiplayer AI needs to move past the group chat phase. Chats are chaotic, everyone sees everything (creating social & management issues), and they're bad for long projects. Worse, they keep AI trapped inside existing teams, when its real promise is working & helping across them.
Andy Coenen retweeted
hey. here's a little demo of the most meaningful improvement to my productivity since speech-to-text.
Andy Coenen retweeted
planning is overrated
just take the shortest term action that is in the general direction of your goal
reality is a super high dimensional space, so this basically always works
cheap unlimited intelligence will force us to rethink basically every single aspect of society
“Let the agents do what they do best and let you focus on what makes you irreplaceable: judgement, direction, and ownership”
Give your agent a way to verify its work. This can dramatically improve the quality of output.
With this process in place, Claude Code was able to write 90%+ of its code itself. When generating software at a faster rate than you can review it, the bottleneck shifts to proving that it works. @bcherny and team invested enough in verification, we've been able to move fast while maintaining stability.
Code reviews aren't going away, but you need to think of it differently. One human reading through line-by-line is no longer going to work as a gate, it needs to evolve into a multi-layered verification system, built with fast deterministic feedback loops for your agents.
This is where e2e tests, feature maps (as @poteto has been advocating), property tests and multi-agent adversarial reviews come in. These all provide opportunities to catch errors early and fix them before things get out of hand.
You don't have enough eyeballs for the amount of code being written so your focus needs to be on product intent, risk assessment and evaluating evidence rather than reviewing code. Delegate the code checking work to your automated tests, scale your approvals based on risk, and turn your test failures into living evaluations. This approach helps maintain speed without losing reliability.
Let the agents do what they do best and let you focus on what makes you irreplaceable: judgement, direction, and ownership. We tried to capture how we handle code review and verification in this writeup. Hopefully it's helpful.
AI labs are completely missing the mark by training models to build massive finished products end-to-end. nobody wants this and it's fundamentally misaligned
data. its always been data
I've been thinking a lot about data bottlenecks, and I'm not sure who else is screaming this from the hilltops. I feel like I'm taking crazy pills looking at the jagged frontier thinking it's smooth if you order by data volume.
the spikes in the jagged frontier are tasks with tons of data (either static or simulated; think math, programming, games), and the valleys are tasks with little or no data (cooking a lasagna, olfaction, manufacturing a passenger train). I don't know of any exceptions.
if you order the jagged frontier by data it looks like a spiral frontier. models are superintelligent where we have enormous data, and progressively less so as we have less data. some tasks may draw data from static lakes of data, some may be from limitless waterfalls from simulators that convert compute into data. but ultimately, intelligence on a task is proportional to the data absorbed by the model for that task.
scaling laws are real. if you are not bound by compute or algorithms (increasingly we are not), then intelligence is proportional to data. since we don't have a very good notion of task similarity, it's often difficult to know if data for one task will help you with another, without training a model and seeing. but, it doesn't seem like chess data helps much with math performance and vice versa. so when I refer to data, I refer to data that is task-relevant.
maybe we call this the 'no shortcuts hypothesis'. all the rules-of-thumb about learning are at the root, the same -- Malcom Gladwell's 10,000 hours, Moravec's Paradox, Scaling Laws, the Bitter Lesson, Age of experience. If your learning system (model, brain, whatever) is unconstrained by compute, intelligence is proportional to accumulated experience. no shortcuts. want to be superhuman at chess? play all the chess, more than the whole species combined (good thing a model can do that in 24 hours). want to be superhuman at math? build a math simulator and lock the model in it and run it faster than reality. superhuman at coding? first, collect all public examples of humans writing and responding to code ever, make the model learn from that first, and then lock the model in a simulator where it fixes bug after bug, faster than realtime.
superhuman at image generation? no shortcuts. but here, we don't have a simulator. so now we're moving around the spiral frontier away from closed-loop into large static lakes. still big enough to be superhuman though, which also applies to video, text, and audio. oh! and once you have a superhuman model, you can use it to bootstrap other tasks by making a superhuman model a judge on other tasks.
let's keep marching along the spiral frontier, out of the realm of data abundance, simulators and large static lakes.
biology? no shortcuts. go get a ton of data. and be freaking careful! you can get bad data in biology pretty easily. tilting a well plate 5 degrees the wrong way can ruin an experiment. so make sure you understand the domain, otherwise you're going to have to filter a lot out.
robotics? no shortcuts. get the data. if you can build a simulator, do it, but be mindful our simulators don't have perfect correspondence with reality. so, build an arm farm. fold 1M shirts, put 1B balls in cups. play ping pong forever.
olfaction? no shortcuts. smell the world. build a laboratory of chemical sensors, an international network of trained sensory panelists, a scent production factory, and link them all together. make millions of samples. smell millions of samples. take billions of chemical sensor readings.
I totally buy super intelligence as a concept because I feel it, I use it for coding, the breakthrough math results speak for themselves. I see RSI coming, because how could an all-digital task like ML research and model building not fall to the current paradigm? But the notion that AI will come for everything, all at once, irrespective of domain, feels silly, because there are no shortcuts. if you want a specific form of intelligence, you have to get the data.
looking at you @willdepue, I feel like we're both seeing the same thing
Andy Coenen retweeted
Finally people waking up to how trash gpt image is at pixel art.
Use Retro Diffusion, I've been building it for 4 years, its best-in-class, trained by a professional pixel artist, and designed for all different kinds of game art.


























