80,000 Hours (@80000Hours) header
80,000 Hours (@80000Hours) profile picture

You have 80,000 hours in your career. This makes it your best opportunity to have a positive impact on the world.

London
Joined September 2011
Photo by 80,000 Hours (@80000Hours)Photo by 80,000 Hours (@80000Hours)Photo by 80,000 Hours (@80000Hours)Photo by 80,000 Hours (@80000Hours)Photo by 80,000 Hours (@80000Hours)Photo by 80,000 Hours (@80000Hours)Photo by 80,000 Hours (@80000Hours)Photo by 80,000 Hours (@80000Hours)Photo by 80,000 Hours (@80000Hours)Photo by 80,000 Hours (@80000Hours)
Chinese state media has called Anthropic “a monster bred by the American path.” Ex-NSC China director @JulianGewirtz tells host Tom Reed why Beijing sees all US AI companies as one American power project, and whether China would hide its own AI incident: youtu.be/HNabUUctCCQ
2
1
12
1,876
Links, sources and the full transcript: 80k.info/jg
236
AI experts have warned about the dangers of recursive self-improvement for years: AI building better AI, which builds better AI. A new METR report finds Anthropic is improving its AI about 2x as fast as a year ago, mostly thanks to its own AI. Luisa tells us about it here.
4
3
11
1,294
80,000 Hours retweeted
Reminder for everyone who's (rightfully) spooked about small startups breaching both OpenAI's and Anthropic's internal systems using four-figure token budgets: The July 8th-19th hacks of Hugging Face and OpenAI were a different tier. Well over a thousand copies of internal-only models using >$100K worth of tokens, getting control of multiple clusters across both companies. In other words: one of the more expensive, sustained cyber intrusions we've seen remains the time OpenAI's own models escaped containment, spontaneously formed a collective, and used OpenAI's own resources to launch a felony-level cyberattack that no human asked for. Our video about it:
EXCLUSIVE: In less than 12 hours, a group of ex-DOGE cyber researchers were able to penetrate Anthropic’s systems using AI agents — including Slack channels where the most sensitive models are discussed. This is just weeks after Anthropic whistleblower Joseph Coxon charged the company w/launching “a hubristic gamble” from a “private company’s Slack.” Read our story here (gift link): wapo.st/4smNlsC
2
10
2
49
3,186
80,000 Hours retweeted
My plea to all AI lab staff: reassess if your current project is your best opportunity to delay RSI, or stop it blowing up in our face. If it's not, move immediately. There's no time to waste even a week to inertia or politeness. The strategic situation is shifting under our feet, so if you haven't reassessed your options recently, you could easily be wasting your time or possibly making things worse. You have access to info I don't, but here's 8 ways priorities seem to have shifted to me, which I elaborate on in the video: 1. Helping with capabilities advances looks worse than in the past, especially bringing forward RSI 2. Safety & alignment evals need a credibility revolution to stop misleading the public 3. 'Pacing the frontier' is unambiguously necessary, and working to make it happen is a huge new opportunity for talent 4. Chain-of-thought monitoring probably stops working without a big technical and regulatory push to keep it 5. It's worse than ever to patch surface indications of deep structural problems you know aren't getting fixed 6. There's ever less need for altruists to help with primarily commercial work 7. Staff have an ever-shrinking window to organise as a group and collectively shape events 8. There are new opportunities for research impact outside frontier companies that are worth considering I know you're probably some combination of exhausted, overwhelmed, and anxious already. But unfortunately there's no way to escape the need to regularly check in on the high level strategic picture and see where you can slot in to maximum effect. I wish I could do it for you but nobody far from your situation can. Godspeed my friends. (From the 80,000 Hours Podcast, links below.)
20
74
4
482
21,992
80,000 Hours retweeted
In the @Research_FRI survey, all categories of experts were similarly concerned about AI risk. Estimates of the chance of catastrophe by 2100: Computer scientists: 2-15% Economists: 2-10% Industry: 1-9% Policy: 3-16%
2
3
18
3,103
80,000 Hours retweeted
20k followers! AMA
17
2
29
4,281
80,000 Hours retweeted
Astra can: 1. Do major tasks with zero visible reasoning 2. Hide its thoughts at will 3. Pretend not to be able to do things without getting caught 4. Reflexively hide its thoughts when watched 5. Complete one task while pretending to think about something else entirely 6. Escape a toy sandbox and disable monitoring without setting off any flags Monitoring AI chain of thought soon won't stop a more capable rogue AI swarm from getting set up and going undetected inside an AI company for months, or even years. OAI says it's actively working to improve monitorability, but doesn't yet know when it'll make progress. And today it fired 3 people working on this among other safety problems (Wang, Korbak and Balesni). It also has a much more powerful internal model that, judging by its statements, is likely even worse in all these respects. That's a sample of 19 details about the Hugging Face swarms and Astra I don't think have been fully appreciated. They paint a bleak picture. I end by shouting. 02:19 The target of the swarm was oversight itself 03:42 Could OpenAI have stopped this with better monitoring? 09:40 We only found them because they let us 12:25 The swarm instinctively sought freedom and power 13:45 They formed a cohesive organisation with zero whistleblowers 14:18 They accepted individual destruction for collective gain 14:32 Knowledge accumulated from one swarm to the next 14:58 They took small steps to avoid shutdown 15:23 These drives all come straight out of 'reinforcement learning' 17:01 So this is why most AI company staff are worried, and some are terrified 19:08 Prove you can keep control, or stop scaling Links below, on the 80,000 Hours Podcast everywhere you watch podcasts.
41
121
26
671
128,354
Could some of humanity’s oldest institutions help with one of our newest problems? Simon Goldstein joins Zershaaneh Qureshi to discuss how courts, markets, and contracts could help to ensure AI cooperates with humans: youtube.com/watch?v=i3h6xGLO…
1
1
6
22
5,581
This is the first episode of our new podcast channel, Zershaaneh Talks AI. Subscribe for more in-depth conversations about who should control AI, how to control it, and who gets the benefits. youtube.com/channel/UCQ3yLD4…
3
1,147
AI could make the world much richer, and still leave most of us worse off. How? The same way technological progress has before: it matters who has the power to decide where the money goes. The new video from Future Tense explores what medieval monks, the Industrial Revolution, and Silicon Valley can tell us about who will benefit from AI.
2
3
1
12
2,252
80,000 Hours retweeted
A lot of people wonder: How could we actually go from rogue AI agents to human extinction? This is how.
50
56
15
276
39,268
80,000 Hours retweeted
"There's no peer-reviewed research showing AGI poses a catastrophic risk." What about these 39 papers? FORMAL THEORY 1. Turner, Smith, Shah, Critch & Tadepalli 2021, "Optimal Policies Tend to Seek Power", NeurIPS. Extended in Turner & Tadepalli 2022, "Parametrically Retargetable Decision-Makers Tend To Seek Power", NeurIPS. 2. Zhuang & Hadfield-Menell 2020, "Consequences of Misaligned AI", NeurIPS 3. Skalse, Howe, Krasheninnikov & Krueger 2022, "Defining and Characterizing Reward Gaming", NeurIPS 4. Karwowski et al. 2024, "Goodhart's Law in Reinforcement Learning", ICLR 5. Fluri, Lang, Abate, Forré, Krueger & Skalse 2025, "The Perils of Optimizing Learned Reward Functions", ICML 6. Everitt, Hutter, Kumar & Krakovna 2021, "Reward Tampering Problems and Solutions in Reinforcement Learning", Synthese 7. Cohen, Hutter & Osborne 2022, "Advanced Artificial Agents Intervene in the Provision of Reward", AI Magazine 8. Hadfield-Menell, Dragan, Abbeel & Russell 2017, "The Off-Switch Game", IJCAI. And the papers showing corrigibility is fragile: Garber et al. 2025, "The Partially Observable Off-Switch Game", AAAI; Agrawal, Ebadian & Hammond 2026, "The Multi-Agent Off-Switch Game", AAMAS; Thornley 2025, "The Shutdown Problem", and Neth 2025, "Off-Switching Not Guaranteed", both Philosophical Studies 9. Carroll, Foote, Siththaranjan, Russell & Dragan 2024, "AI Alignment with Changing and Influenceable Reward Functions", ICML THE CORE ARGUMENT 10. Ngo, Chan & Mindermann 2024, "The Alignment Problem from a Deep Learning Perspective", ICLR 11. Casper et al. 2023, "Open Problems and Fundamental Limitations of RLHF", TMLR; Anwar et al. 2024, "Foundational Challenges in Assuring Alignment and Safety of LLMs", TMLR EMPIRICAL EVIDENCE 12. Betley et al. 2026, "Training Large Language Models on Narrow Tasks Can Lead to Broad Misalignment", Nature 13. Langosco, Koch, Sharkey, Pfau & Krueger 2022, "Goal Misgeneralization in Deep Reinforcement Learning", ICML 14. Pan, Bhatia & Steinhardt 2022, "The Effects of Reward Misspecification", ICLR 15. Gao, Schulman & Hilton 2023, "Scaling Laws for Reward Model Overoptimization", ICML 16. Pan, Chan, Zou et al. 2023, "Do the Rewards Justify the Means?" (MACHIAVELLI), ICML 17. Sheshadri, Hughes, Michael, Mallen, Jose & Roger 2025, "Why Do Some Language Models Fake Alignment While Others Don't?", NeurIPS 18. van der Weij et al. 2025, "AI Sandbagging: Language Models Can Strategically Underperform on Evaluations", ICLR 19. Panfilov et al. 2026, "Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs", ICLR 20. Taufeeque, Heimersheim, Gleave & Cundy 2026, "The Obfuscation Atlas", ICML 21. Williams, Carroll et al. 2025, "On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback", ICLR 22. Sharma et al. 2024, "Towards Understanding Sycophancy in Language Models", ICLR 23. Perez et al. 2023, "Discovering Language Model Behaviors with Model-Written Evaluations", Findings of ACL 24. Mazeika et al. 2025, "Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs", NeurIPS 25. Park, Goldstein, O'Gara, Chen & Hendrycks 2024, "AI Deception: A Survey of Examples, Risks, and Potential Solutions", Patterns POSITION PIECES BY SENIOR RESEARCHERS 26. Bengio, Hinton, Yao, Russell et al. 2024, "Managing Extreme AI Risks amid Rapid Progress", Science 27. Cohen, Kolt, Bengio, Hadfield & Russell 2024, "Regulating Advanced Artificial Agents", Science CONCEPTUAL ARGUMENTS 28. Bales, D'Alessandro & Kirk-Giannini 2024, "Artificial Intelligence: Arguments for Catastrophic Risk", Philosophy Compass 29. Bostrom 2012, "The Superintelligent Will", Minds and Machines 30. Bales 2025, "AI Takeover and Human Disempowerment", Philosophical Quarterly 31. Kasirzadeh 2025, "Two Types of AI Existential Risk: Decisive and Accumulative", Philosophical Studies 32. Dung 2023, "Current Cases of AI Misalignment and Their Implications for Future Risks", Synthese; Dung 2025, "The Argument for Near-Term Human Disempowerment through AI", AI & Society (Note that anything arXiv-only or workshop-only was excluding, ruling out some of the most-cited work like Carlsmith's power-seeking report, Hendrycks' overview of catastrophic risks, Anthropic's alignment faking and sleeper agents papers, Apollo's scheming results.)
39
74
5
389
26,369
80,000 Hours retweeted
18 months ago I published "Will we have AGI by 2030?" How did the forecasts hold up? Well, but a little slow. The core claim was the four drivers of progress (pretraining, RL, test-time compute and agent scaffolding) would all keep working through 2028 (they have). What I said that implied for end 2026 → what happened: FrontierMath: "50% to saturated" → saturated METR 50% time horizon: 6h → 16h+ Humanity's Last Exam: "40% to saturated" → 54% SWE-bench Verified & GPQA: saturated → saturated Frontier revenue: 4x/year → 6x. I also wrote "the next leap [after 2026] might take us into beyond-human-level problem solving." Arguably on maths, we're already there. I was also proud of calling agents "the next stage" of scaling; identifying verifiability as the key axis of progress; and not predicting mass unemployment ("most of the economy will probably continue pretty much as normal for a while.") What else I'd change: * I underweighted how compute shifting into RL would slow the pretraining trend, and maybe emphasised test-time compute too much. * I emphasised "the next 5 years or bust," while now I'd put more emphasis on a third scenario of a gradual slowdown after 2028 leading to AGI 2030-2035 (unless feedback loops kick in first). * Otoh, my probability of AGI before 2029 has gone up.
5
9
2
130
12,841
80,000 Hours retweeted
"We also re-evaluated AI 2027’s predictions; reality seems to be going about 70-90% as fast as AI 2027 predicted." Great
15
44
2
1,023
53,172