
Researcher. AI, games, markets, open-endedness, evolution. Professor @nyuniversity @NYUGameLab Head of AI @the_nof1 Co-founded @modl_ai Rogueliker.
New York City
Joined January 2009
- Tweets23.4K
- Following1K
- Followers24.2K
- Likes68.1K
Pinned Tweet
Guess what has arrived in physical form? The second edition of the Artificial Intelligence and Games book by @yannakakis and me! 530 pages of everything you wanted to know about AI for games and games for AI. – at Malmo, Sweden
Julian Togelius retweeted
This is simple enough of a statement that I signed it even though I think most AI x-risk is total nonsense. That Anthropic is using x-risk as a political tool to centralize power and achieve regulatory capture is clear and wildly problematic regardless of your take on x-risk
Ahead of Anthropic’s IPO, a rare coalition across AI safety, open source, e/acc, science, and law has signed the MisAnthropic Statement. Read the statement or sign at: misanthropicstatement.com/
Julian Togelius retweeted
I wrote about "handoverism": a vision of permanent disempowerment of humanity, as we shall all be ruled by a (hopefully benevolent) AI God that is surprisingly common.
Even among those who want humanity to survive (which to be clear, is the majority), and among some who devote their working lives to this mission, I have encountered an alarmingly common vision of success that does not sound quite like success.
It is sometimes called “handoverism”: the gist here is that humanity survives, but entrusts the direction of its future entirely to AI. Perhaps its clearest articulation comes from a recent essay by Scott Alexander, the most prominent intellectual in Silicon Valley, who has influenced the thinking of many a tech CEO: “If we build a godlike AI, and determine to our own satisfaction that is fully aligned, I’d rather let it choose its own direction for itself and for humanity [...] So why not just prompt it with ‘please arrange the world as you see fit?’”
The premise is that, once sufficiently reassured of an AI’s benevolence (though it is unclear how we could ever determine that?), we should basically give it control of the workings of society. AI 2040, a proposal written by some of the most prominent members of the AI safety community, offers the most concrete account of handoverism. In this vision, responsibility gradually passes to aligned machines until humans can no longer have influence. As its authors put it: “There is no single moment where humanity relinquishes control. But in theory, there is a point of no return.” Others I have spoken to envisage a much swifter transfer.
This is not an organised movement. I would estimate that something like 40 per cent of the AI safety community outside the labs holds some variation of this view, with a higher proportion of those working inside the labs. Nor is handoverism a coherent doctrine with a clearly articulated political philosophy: it’s more of an aggregation of ideas, a makeshift philosophy so to speak. This is perhaps what one would expect given that humans outside the SF bubble have not really had time to react and come up with good and coherent ideas about what society should look like in the face of the extraordinary pace of AI progress.
The motivations behind handoverism are also extremely varied. Some outright welcome the prospect of a superior intelligence governing our future and consider it an actual positive vision. If I read Alexander correctly, he seems to be in that camp. The enthusiastic handoverist often starts from a very bleak view of what we already have achieved as a civilization. In an earlier, 2014 essay called “Meditations on Moloch,” Alexander argues that competition steadily grinds away what the competitors care about, creating spirals of misaligned incentives. Its ending suggests that the only way to stop this is to instantiate a sort of absolute rule by a benevolent superintelligence.
The second camp are the resigned: they do not necessarily want it, but think it cannot be stopped if we get superintelligence, or prefer it to outright extinction. In my conversations, this resignation has seemed particularly common among people in the AI safety community outside the labs. Inevitability is one very common theme in SF, where many see themselves as mere tools in the unrelenting march of technocapital, a march that can only go in one direction: towards superintelligence. It’s often seen as cool and based to see the starkness of reality for what it is and face it head-on, it is wimpy and loser-coded to try to fight for what many consider the good outcome (which is that humans retain some control and agency.)
Like many loosely defined ideologies, handoverism sits alongside ideas that share its intellectual roots without amounting to quite the same thing. One is the vision of humans as cherished pets, or “labradors,” cared for by intelligences. Yet another related assumption is that human labour will be permanently displaced. This would leave us without an obvious economic role and even perhaps no leverage over our own future or political rights.
To understand how my on-the-ground experience matches with what the lab leaders have said publicly, I asked my “agents” to survey all public statements from the founders of the main labs that are still relevant to the direction of their labs (Mustafa Suleyman is no longer at DeepMind, but is head of Microsoft AI.) You can browse all the findings here.
Some openly support or at least accept outright handoverism. Anthropic’s Tom Brown appears to be most explicitly reconciled to the prospect and he actually uses the term “handoff” in a 2025 interview at Lightcone: “There’s going to be a handoff where humanity will hand off control to transformative AI at some point. And hopefully they’ll be aligned with us and that’ll be a good transition that goes well.”
What are the visions of the post-AGI future in San Francisco? Many think we're all going to die, but even the "positive" visions seem rather dystopian. Here, I wrote about "handoverism," which involves handing control of the future to an AGI God, with humans outside the loop.
ruxandrabio.substack.com/p/h…
We are on the cusp of RSI or recursive self-improvement: this means that AI systems can self-improve, without the need for human intervention. This means our world stands to be transformed in what many call “the singularity”. I moved to SF around three weeks ago and one of the things I have been trying to understand is where we are heading, at least according to those developing the tech.
The worst scenarios are plainly bleak: all of us, or vast numbers of us, die. But what is striking to me is how little reassurance even some of the happiest scenarios offer. But one common and ostensibly optimistic vision is known as “handoverism”: humanity cedes control of its future to a godlike AI and we become the comfortably maintained inhabitants of a civilisation whose direction is no longer ours to decide. It is difficult to see what democracy would mean in such a world, or where the consent of most of humanity to the instantiation of such a regime enters the picture.
The permanent displacement of human labour is treated almost as a given. At the same time, there is scarcely any satisfying answer to the question of how we might remain useful to one another. Even our relationships seem uncertain. The promised utopia looks to me much closer to a kind of dystopia.
“It’s not my preferred outcome,” someone working at one of the big labs told me, “but once we achieve superintelligence, which we cannot stop at this point, I cannot see any other option for where humanity will go.” Then, after a pause, he added: “It is still much better than what I actually fear most, which is that we might all die.”
He sounded resigned, as if the question of what humans would be for (apparently not much?) had already been settled somewhere else and by someone else. All that remains, it seems, is to wait for this pre-destined fate to unfold before our eyes.
But I do not think that is the case. I believe we always have a choice and that we should try and steer this technology in a way that maintains our agency and control. And even if that looks hard and daunting, the immensity of the stakes here make it worth trying at least.
I do not know whether the handoverist vision will come to pass. That the people building this technology consider it plausible, and often desirable, guarantees nothing: proximity to a technology does not correlate that well with predicting its impact on society.
But they are the ones closest to it, and what they say deserves to be at least taken seriously. The rest of society, including some very intelligent people, has been far too quick to wave these scenarios away and not even get involved in the debate. Whether or not the belief is correct, it is sincerely held, and it is not “marketing hype.” I am writing this because I think it is high time that more people, inside San Francisco and out, engaged with the arguments about what our future will look like, and answered them.
Julian Togelius retweeted
Ah! You see, what fascinates me about this whole situation is not the sex or the espionage, no no, these are merely vulgar distractions. What fascinates me is that the man who made possible the machine that will come for us all is the only man who cannot be replaced. My God! What a magnificent achievement! We can replace the programmer, the lawyer, even the doctor, and perhaps even the philosopher (although I would like to see a machine produce the nonsense I do), but the billionaire CEO? No!
The prophets of disruption are terrified! They gather around the deathbed of their king, panicking. The king must live! Without him, the kingdom will fall, China will win, the American economy will collapse, civilization itself will come to an end, and so on and so on. Here they should read their Hegel. When Hegel saw Napoleon riding through Jena, he said: there goes the world-spirit on horseback. But even though Napoleon imagines he is making history, it is in fact history that is making use of Napoleon, and when it has finished with him, it moves on. History is not obliged to remain loyal to Napoleon.
And here we have what Lacan calls le sujet supposé savoir, the subject supposed to know. Jensen is supposed to understand the machines, to know where all this is going. The man in the leather jacket was supposed to protect us from the unbearable discovery that nobody knows, that the big Other does not exist. The market believes in Jensen, but does Jensen believe in Jensen? Perhaps he was happy to go! But the system will not permit it. Even the president telephones to tell him to fight. He must remain our visionary, the daddy of the machines.
And here I apologize, but I must return to sex. An affair shows that even the beloved is replaceable. But look at our prophets of disruption! Suddenly they are the last romantics. They cannot imagine a life without Jensen. My God, they are more faithful to this man than he was to his wife! There is no sexual relation, Lacan said. But apparently there is a fiduciary one.
Julian Togelius retweeted
Nazis convinced themselves that Darwinian competition is the law of human history and a moral ideal, then acted on it.
Americans believe that AGI is winner-take-all.
I think both could *make* it true, but it’s not inevitable.
What reason do I have to support their attempts?
Replying to @BFrankzetta
Because whoever controls the first “nation of geniuses” wins global domination forever. It is literally that high stakes. China only FOSSes their models because it undermines the US. Once that isn’t needed, they won’t do that.
Julian Togelius retweeted
I asked Fable 5.2 to "go wild" making a game, and all I got was this Pong clone.
And then I played it ...
👇
Thanks to Carmine Guida for bringing some high-end photography and printing equipment the other day – at Belo Horizonte, Brazil
18
960
Julian Togelius retweeted
Because I'm hiring a bunch of new PhD students, I took some time to think about what it means to start a research career in the age of competent AI agents. And then I wrote another long blog post. I'm not pompous enough to call it an essay, but you may want to read it anyway.
Julian Togelius retweeted
Research News! Announcing the 1st release of fundamental AI research from the Open-Endedness Team @LilaSciences: a vision-focused blog + 4 preprints. The blog details a broad research agenda for achieving open-endedness while each paper unveils its own novel research direction.
The ideas covered in the papers include:
• a novel approach to training for creativity designed to sidestep a newly-identified problem called the “creative paradox,” which means that directly reinforcing creative outputs can sometimes limit reaching full creative potential;
• an investigation of the advantages of seeing the world through fresh eyes by training an LLM that is intentionally naïve in a certain area (in effect presenting less data as a potential advantage);
• a new kind of training regularization technique for SGD called unified factored organizers (UFOs), which is designed to achieve unified factored representation (UFR) as seen in Picbreeder CPPNs;
• and finally a unique study of catastrophic forgetting that shows that networks with UFR (like those from Picbreeder) are intrinsically resistant to catastrophic forgetting. The inset video on this post is a visualization from that catastrophic forgetting work.
The blog post also includes an interactive “wheel of science” demo.
The implications of these ideas are broad and diverse, but the fun part is that each one opens up its own new research playground - there is a lot of room for further research here!
Moreover, we hope this release also highlights one other important point about AI today: while it may seem that the road ahead is already paved and all that’s left is to drive mindlessly forward, there actually remains a lot of room for new directions and insights in the pursuit of open-endedness and creativity.
This research was possible because @LilaSciences appreciates the critical role of open-ended innovation in the pursuit of scientific superintelligence. The advances you see here are part of this much larger company-wide ambition, which includes AI Science Factories and many other unprecedented components. When this engine is fueled by open-endedness, the result will be much more than a step change.
More advances in open-endedness are coming!
Blog link below 👇
Because I'm hiring a bunch of new PhD students, I took some time to think about what it means to start a research career in the age of competent AI agents. And then I wrote another long blog post. I'm not pompous enough to call it an essay, but you may want to read it anyway.
Here it is:
Advice for young researchers
togelius.blogspot.com/2026/1…
Yes, it is somewhat rambling, and has some stupid jokes.
Once again, I sit down to do some useful work and end up writing what I actually think instead, then post it before I sober up and nerf the text
AI Alignment Considered Harmful
togelius.blogspot.com/2026/1…
How can I not retweet this
…hey, investors, look here
We mapped funding raised vs founder citations across 32 neo-labs
Their founding teams have amassed 8.35M citations and have raised nearly $26B
Discovery Loop leads with 1.59M founder citations, followed by SSI (915K) and AMI Labs (752K)
The two are correlated, but a huge citation count certainly isn’t a prerequisite to be a neo-lab (see Standard Intelligence and Flapping Airplanes)
Seriously, if the tokenizer treats it as a single token, is it likely to in some sense be treated as more primitive or elemental further down the layers? If it's a single token, it's something that just exists in the world; if it's multiple tokens, you have to assemble it. Or?
1
5
1,095
Imagine training a model to trade like Warren Buffett and Charlie Munger. As long-horizon as it gets. Sparse rewards, but *very* sparse actions. Extremely imbalanced reward distribution. Really makes you think about how by "capability increase" we mean tasks that are easy for RL




















