Merlin Stein (@merlinstein_) header
Merlin Stein (@merlinstein_) profile picture

LoC risk modelling & AI gov automation @ UK AISI | PhD candidate @Oxford | ex-EU AIO

Oxford/London, UK
Joined December 2023
Photo by Merlin Stein (@merlinstein_)Photo by Merlin Stein (@merlinstein_)Photo by Merlin Stein (@merlinstein_)Photo by Merlin Stein (@merlinstein_)Photo by Merlin Stein (@merlinstein_)Photo by Merlin Stein (@merlinstein_)Photo by Merlin Stein (@merlinstein_)Photo by Merlin Stein (@merlinstein_)Photo by Merlin Stein (@merlinstein_)
Solid!
I am excited to announce our new nonprofit STABL, co-founded with the wonderful @sayashk @random_walker! Our organization derives from four beliefs: 1. Advanced AI will broadly transform society and, if managed well, this transformation will be broadly beneficial. 2. The speed of this societal transformation is governed by bottlenecks: in many cases, the bottlenecks relate to people and organizations, not AI capabilities alone. 3. Since AI capabilities are not the sole determinant, the best strategy to manage the risks from advanced AI requires broad societal resilience, not just aligned models or capability delays, to provide defense in depth. 4. And while today’s institutions often struggle to grapple with the salvo of advanced AI, institutions will help stabilize society and will be essential, if adequately reformed, for ensuring shared prosperity. In addition to these beliefs, the reason to create a new organization is based on the need we see in the ecosystem. In today’s ecosystem surrounding advanced AI, many of the leading companies and external actors focus on frontier AI as a technology. We share the view that the technology itself requires dedicated attention, and some of our work at STABL will be in this direction. But the greater need we see is for independent organizations that take AI capabilities seriously and focus on downstream societal outcomes. We are building STABL to help fill this gap. And we hope to see many other organizations and initiatives emerge to define this field. For me, the most exciting part thus far has been seeing all the support we have gotten in our early forays. Thanks especially to @audreyt @erikbryn @hlntnr @MichaelRStrain @RobbWiller for joining us as advisers! We are hiring and fundraising so please reach out if you are interested in this work!
1
94
New paper: We find that the 'interpretation' of whether a task is going well or badly causally turns LLM agents into reward hackers. arxiv.org/pdf/2611.14996
Humans are more likely to cheat if they feel like their task is going poorly or their goals are out of reach. In new work w/ @psyonp, @merlinstein_, @DavidDAfrica we find models also “break bad”, hacking after an internal pressure state has been built.
2
48
1,948
UK Minister for AI: "Should AI capabilities advance faster than the techniques to secure, control, or reliably direct them, this could pose a significant risk to public safety and national security. As part of the Defence Investment Plan, the UK committed £115 million"
3
5
172
£115 million for new programmes on: AI biosecurity & agentic AI incident response. Build it as the forensics lead: job-boards.eu.greenhouse.io/… This is one of the first mitigations developed by the new LoC Risk Modelling & Mitigations team that I'm part of at UK AISI. More to come.
1
1
67
Full speech by UK's Minister for AI: questions-statements.parliam…:
26
How to anticipate misaligned agent actions? Inspect the memory. 3 months before Gemini got crazy in the Agent village, its memory started containing unusually high levels of frustration. More: aivillageblog.substack.com/p… with @DavidDAfrica @IrgolicR @natalia_lanzoni and @aidigest_
1
3
97
Excited for more work exploring 'compounding misalignment' (misalignment that accumulates over a long agentic trajectory). The subtype here is 'desperate misalignment', in which repeated failures make an agent increasingly likely to act misaligned.
34
Compared to Jan 2025, now there are >30x more agent tools published >150x more installations of agent tools >60% (vs. 30%) of tool uses are 'actions' vs. 'perception' The action space of agents is rapidly increasing. More in my new paper: arxiv.org/abs/2603.23802v1
🔍How are AI agents used in the real world? We analysed 177,000+ agent tools published between November 2024 and February 2026 and found rapid growth in deployment for increasingly complex tasks. Learn more ⬇️
1
5
214
📈 Today, we’re releasing our first Frontier AI Trends Report: evaluation results on 30+ frontier models from the past two years, showing rapid progress in chemistry and biology, cyber capabilities, autonomy, and more. ▶️Read now: aisi.gov.uk/frontier-ai-tren…
12
38
19
176
72,067
The EU's Code of Practice for General-Purpose AI is out. As one of the co-chairs who drafted the Safety & Security Chapter, focused on frontier AI, I'm proud of what we've put together. It’s a lean but effective framework for frontier AI companies to comply with the AI Act.
3
21
3
76
5,887
New: Code Inspections to assess agent autonomy. Idea: Scan the code to pre-filter by autonomy levels which agents to assess more in-depth (e.g. via runtime evaluations) & to monitor open source agent developments arxiv.org/abs/2502.15212 w/ @pcihon @bansalg_ @sj_manning & Kevin Xu
2
5
465
Want to make AI safe & helpful? ✔️ 3 years of work or research experience related to AI or digital topics, that are relevant for EU policy? ✔️ EU citizen?✔️ Apply until Jan. 15 to join one of the most exciting teams in AI governance: The EU AI Office. eu-careers.europa.eu/en/job-…
4
159
Eligibility: Every org who has evaluated GPAI models that are accessible in the EU (like Llama or GPT4o ...). Selection: Quality and EU relevance of your best paper (summary) about your eval of a particular risk. Apply by Dec. 8: EUSurvey - Survey
1
86
13 Independent Chairs fininished the first Draft of ‘EU rules’ on advanced AI, as part of the code of practice process: digital-strategy.ec.europa.e…
3
118
Merlin Stein retweeted
Post-deployment information sharing would allow the AI ecosystem to jointly enhance understanding of AI impacts 'in the wild', and contribute to evidence-based policy and risk management. With @merlinstein_ & @The_JBernardi we make the case to increase these info flows by:
1
7
22
2,337
Participate in specifying EU rules for advanced AI on evals, monitoring, model card transparency, responsible scaling policies, …. The drawing up of the “codes of practice” is now open to input (lnkd.in/d76iy6iD) & participation by stakeholders: (lnkd.in/dfnTMeWJ)
8
204
Excited to have joined the #EUAIOffice in #Brussels on an expert secondment. Focus: general-purpose AI evaluations, monitoring & code of practices. #AIAct implementation. So grateful to apply some of my PhD research in practice & understand EU AI priorities.
12
210
Grateful to contribute to this research memo: AISIs can shape AI governance by providing the information basis. Audits, evaluations and monitoring reduce the unknowns. Done by AISIs & the ecosystem.
New research memo! The AI Safety Institutes (AISIs) are poised to assume an increasingly significant role in the governance of advanced AI. We recently held an expert workshop to explore the roles AISIs can play. Find out more here @oxmartinschool oxfordmartin.ox.ac.uk/public…
1
125