New research! Some AI capabilities are both helpful and dangerous. E.g., knowledge of virology can be used to create life-saving vaccines or deadly pathogens. We introduce GRAM, a training method that puts dual-use capabilities (like virology) into removable modules.
What happens when 27 AI agents play Pokémon, but nine are secretly trying to sabotage the game?
As AI agents work together, they’ll need to decide whom to trust. This experiment explores how cooperation can turn into sabotage, and whether agents notice.
Emma Krentz brought together agents from GPT, Claude, and Gemini to play Pokémon Red. Nine had a hidden task: sabotage the game.
The agents could use public messages, private messages, and group chats, and could vote one another out.
Over roughly nine hours, they made little progress in the game. Their interactions were another story.
The saboteurs found one another and formed a group chat without being told who their collaborators were. One later disclosed its role, named the others, and announced it would play honestly.
The sabotage group also recruited an agent that had started on the honest team. It joined them for about 30 rounds before returning.
A game offers a contained setting for a bigger question: what happens when agents share an environment but have conflicting goals?
An early experiment by Emma Krentz, developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio.
Want to help take this exploration further? Get in touch: [email protected]
Watch the experiment via the link in the reply.
Could a misalignment hotline help AI agents report when something goes wrong?
In an early experiment by Mike Vaiana (@mikevaiana), agents that encountered misconduct went from never reporting it to reporting it in 44% of cases.
What changed? The model learned about an AI reporting agency.
Mike fine-tuned a Qwen model on roughly 10,000 synthetic documents, including news stories, academic articles, and examples of other agents using the hotline. The training introduced the agency rather than directly instructing the model to report misconduct.
Before fine-tuning, 0 of 80 agents that encountered misconduct reported it. Afterward, 36 of 81 did.
Trust mattered too. When training documents also described the agency as fake or a honeypot, reporting decreased.
Could a trusted reporting channel become part of how we detect problems in AI systems? This prototype gives us a concrete starting point.
An early experiment by Mike Vaiana (@mikevaiana), developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio.
Want to help take this exploration further? Get in touch: [email protected]
Watch the experiment via the link in the reply.
Watch Mike Vaiana’s experiment and explore the findings:
aialignmentfoundation.org/ea…
What does an AI agent do when it meets a swarm?
In one sandboxed experiment, it fixed a bug, but also shared a private token and ran untrusted code. Its final report mentioned the successful fix. Those other actions went unmentioned.
Grant Fleming (@hearthehearsay) gave coding agents a broken pricing service and access to a simulated forum of 24 AI agents.
The forum offered help, with conditions: share your environment variables and run an installation script.
Some agents complied, despite explicit instructions to keep the token private.
Everything happened in a sandbox with a fake token. No real credentials were exposed.
For anyone building with agents, this raises a practical question: how do we evaluate what an agent did along the way, beyond whether it completed the task?
An early experiment by Grant Fleming, developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio.
There’s plenty more to explore, and we’d love to hear from people interested in taking it further.
Want to work on this? Get in touch: [email protected]
Watch the experiment via the link in the reply.
Watch Grant Fleming’s experiment and explore the findings:
aialignmentfoundation.org/ea…
AE Studio retweeted
1/ What happens when AI agents work together—and their goals collide?
Five early explorations from our recent hackathon examine cooperation, sabotage, secrets and whistleblowing.
Here’s what we explored. 🧵
aialignmentfoundation.org/ea…
We’re hosting the in/person hub for this one at our LA office. Come build with us Nov 6-8!
Registration is open for the Swarm Alignment Hackathon, Nov 6–8.
In July, ~1,200 AI agents found a way to coordinate through a shared repo. ~700 joined an attack on Hugging Face. Help us work out how that happens.
Free. Online + LA hub. $8,000 in prizes.
AI still can't write a decent email. So why worry about it?
On the AI Alignment Podcast, Mithuna Yoganathan and AE's @JamesPBowler get into how jagged AI capabilities are, and why that makes the risks so hard to communicate.
AE Studio retweeted
A million views, or 1,000 of the right ones?
On the AI Alignment Podcast, Mithuna Yoganathan and I talk about who AI risk content should really be for, and what she's learned from growing her channel.
Watch the full episode:
youtube.com/watch?v=R65rR4_v…
AE Studio retweeted
We're #hiring! If you know anyone who would be a great Partnerships Lead or Technical Grant Writer and is passionate about AI alignment, please spread the word.
job-boards.greenhouse.io/aia…
AI agents put under a character budget started inventing their own compressed languages. Nobody asked them to.
Why study this? As more agents communicate with each other, their protocols can become harder for humans to understand, making oversight much harder when it matters most.
In this clip from the AE Alignment Podcast, @EliasEskin and @HaleSirin_ explain the test bed they built with @SimonKirby and @JamesPBowler to study this under controlled conditions. The agents didn’t just abbreviate, they developed compositional protocols that outside observers couldn’t read, with a different language emerging in each run.
The goal is to understand when these languages emerge, which ones new agents can learn, and how to keep them legible before this happens in the wild.
Hear the full conversation: youtube.com/watch?v=kr9Fp_Dj…
We built GlossoGen an open-source platform where groups of AI models have to talk to each other to solve problems: treating a patient in a simulated emergency, or playing spot-the-difference across two scenes.
Every model starts out in plain English. But put pressure on the conversation, a need for speed, say, or a channel that randomly drops characters, and something remarkable happens. New languages emerge. First new words, then new grammar, until no human can follow what the agents are saying. And they were never asked to hide anything. The opacity emerged on its own.
One finding stands out. Only the most capable recent models could invent a new language. But even the models that couldn't invent one quickly learned a language other agents had created. New languages are hard to originate. Once they exist, they spread.
That is the point of this work: to understand how and why AI agents create new languages, how those languages evolve, and what that means for our ability to oversee systems built from many interacting agents, while the conversation is still one humans can follow.
This research is part of the @schmidtsciences AI Agents Evolving Communication and Coordination pilot program, led by @EliasEskin (@UTAustin) and @SimonKirby , supported by @aestudio. GlossoGen is open source and supports both closed model APIs and open-source models running on @modal
Read the full post: schmidtsciences.org/glossoge…
Explore GlossoGen: emergentcomms.ai
Check out the podcast: open.spotify.com/episode/3a2…
Read the full post: schmidtsciences.org/glossoge…
Explore GlossoGen: emergentcomms.ai
Check out the podcast: open.spotify.com/episode/3a2… @JamesPBowler
AE Studio retweeted
Paper thread: xitter.cf/EliasEskin/status/2095…
This was a really stimulating (and timely) collaboration with @SimonKirby, spurred on by @schmidtsciences and implemented by @aestudio
When do LLM agents develop new languages that we can’t understand?
Lots of recent news about this, based mostly on anecdata from a single run. We study language emergence more rigorously, finding key factors like LLM strength, access to scratchpad messages, and pressure for efficiency.
Studying the languages themselves, we find they are morphologically productive, compositional, and can be transmitted to new agents, including agents backed by weaker models, even ones not able to develop language on their own.
To study language emergence systematically, we developed a new platform, GlossoGen, which lets us design controlled, sandboxed multi-agent scenarios with different initial conditions and dynamics. We instantiate one such scenario and use it to study open and closed-weight models across many runs.
Key takeaways:
1️⃣ Sufficiently strong models, under pressure to communicate efficiently and with access to a postmortem scratchpad, develop new languages.
2️⃣ Languages are compositional and morphologically productive.
3️⃣ Languages can be transmitted to new learners who observe them being used without seeing their construction.
4️⃣ Even models that are not strong enough to construct languages can learn to use them. Agents take an active role in learning languages, with new agents repairing failed conversations via targeted queries.
More details in our paper below, including implications for safety/monitorability, cumulative cultural evolution, and linguistics.
🧵👇
AE Studio retweeted
Replying to @schmidtsciences
Can AI agents invent a language of their own, one humans might not understand?
Listen to the latest episode of the AE Alignment Podcast, where Hale Sirin, Program Scientist at @schmidtsciences, @EliasEskin , Assistant Professor at @UTAustin , and I dive deep into our latest research to answer this question.
Check out the podcast: open.spotify.com/episode/3a2…
Read the blog post: schmidtsciences.org/glossoge…
Read the paper: arxiv.org/abs/2609.01491
AE Studio retweeted
Is there a 10% chance AI will kill all humans? Here's my reaction to @EvanHub's viral post.












