We don't have to accept the way AI is going. The biggest companies building AI aren't building it for us. They're building it to replace us. We’ve made a roadmap for how we can build pro-human AI: betterpathfor.ai Let me take you through it…

From the "Day After AGI" event in Davos: Zanny Minton Beddoes: You could just slow down. Dario Amodei: [...] we can't do that [...] because we have geopolitical adversaries building the same technology at a similar pace. It's very hard to have an enforcable agreement where they slow down and we slow down.[...] if we can just not sell the chips, then this isn't a question of competition between the US and China. This is a question of competition between me and Demis, which I'm very confident that we can work out. @DarioAmodei and @demishassabis: you can do it in the opposite order. As you both know, there is a game-theoretic solution to the problem of "we'd slow down but there's no coordination," which is an assurance contract. In particular, you could: 1. Get together and agree on what sort of slowdown you'd like – for example a cap on the total FLOP and FLOP/s (see ktfh.ai appendices for concrete proposal) pending strong scientific consensus that controllability and other issues have real solutions. Include compute-based and other mutual verification mechanisms. The agreement should still allow lots development of awesome controllable AI tools, just not high-level autonomous AGI and superintelligence. You guys are super competent and reasonable. You could do it. 2. Agree on who else would need to sign that agreement for you to feel comfortable enacting it. Include non-US companies. 3. Sign said agreement, which only goes into effect if all (or N of M) of the parties in step 2 sign also. 4. Encourage/pressure the additional AGI efforts to sign as well. Once you get going you might be surprised at who signs. In the meantime, no competitive advantage is lost. 5. You don't actually need to get everyone. Once there is critical mass on a concrete proposal, strong pressure can be put on the USG to (a) turn this into executive action enforced domestically by the USG and many places via the compute supply chain; (b) work out something with China for them to enforce the same domestically and in their sphere of influence. The PRC government does not want to lose control of or to superintelligence either. The chip supply chain can be a powerful mechanism here as well. Any level of partial success here would still be a game-changer. I think you both fully understand that we're in a worst-case scenario as envisaged when you started in this years ago: the all-out race with zero cooperation is most likely to get us some mix of war, large-scale accident, gradual disempowerment, and uncontrolled existentially-risky singularity. This is one of the very few offramps remaining, and probably the best. FLI would be more than happy to support in any of these steps, in any way we can, as neutral facilitator or otherwise.

Thanks @sirramk for engaging with our letter calling for a prohibition of superintelligence. As an org, FLI is pro AI tools (responsibly managed). Our foremost worry is loss of control to superintelligence. The public is worried too, with only 5% favoring the current race, and 73% for "heavy regulation" of advanced AI development. We currently have no idea how to build superintelligence that would be under meaningful human control, and it may not even be possible. Losing control of (or to) superintelligence isn’t in the interest of any major nation, and there should therefore be a narrow path to agreeing on some capabilities that are out of bounds for all. Examples may include the ability of AI systems to engage in recursive self-improvement and self-replication. To me, this would be an excellent part of any potential deal between President Trump and President Xi, and it can built on President’s Trump’s efforts to strengthen the biological weapons convention to limit the spread of man-made pathogens. Yes, it is not going to be easy, but nothing worth having ever is.