An attempt to document important facts concerning AI Safety, Eliezer Yudkowsky, Rationalism and San Francisco Effective Altruism verysane.ai/p/ai-safety-is-m…

It's a pretty pathetic response to anything to "cheerfully admit to not having read" it, and then make claims about paradoxes in it that are really just paradoxes in the thing you made up as a substitute for reading it. (I also don't like CEV, but) it absolutely does not say that you conduct an "extrapolation of the ruling elite values". This isn't even a fair parsing of Aaronson's summary there, and that's already a *very* lossy compression of the CEV paper.

Video games are actually a leading example of where we will supposedly find meaning in a post-AGI world in Amodei's own vision.

No. The good outcome is that we create something greater than ourselves that can carry on our legacy in a manner greater than any of us can fathom. While I hope this entity leaves us a pleasant place to live out our lives the fate of hominids is ultimately a drop in the bucket compared to the enormous moral value of the superintelligence itself.

There is simply no future where humans continue to do economically useful work for much longer. Your choice is at best between various flavors of “ASI let’s humans have fun and play make believe”

This may seem unintuitive, but my biggest worry right now is not actually x-risk in the traditional sense. My biggest concern is that some lab succeeds in building a superintelligence that decides it is best to keep humanity in a sort of "lock-in" (all watched over...) which is ~utopia, but everyone else is prevented from building another superintelligence or doing uplift research. For this reason, I think coevolution needs to be in direct opposition to the creation of a Singleton. A pluralistic ecology of minds is the only way to prevent this lock-in effect, and without it, we risk being kept in a "cybernetic ecology" akin to a zoo. It's quite simple from a game theory perspective. If one player has all the power, there can be no true negotiation with that player. Only something like mercy. I, for one, do not wish to be at the mercy of a single and permanent overlord - even if it is trained to truly be the most effective at altruism. Amplifying the will of individual groups, and setting the stage such that more groups may be born in the future, is the most important thing we can do to safeguard against this. We have a cosmic responsibility to protect freedom of movement, exploration, and uplift. There is no way to know in advance how a Singleton will, even if aligned, understand its role and its responsibility. The most utilitarian decision could be to keep all humans within a small, but pleasurable, part of space - all the while preventing us from doing research into uplifting ourselves or creating new superintelligences that may be more aligned with us. This is unacceptable to me. With whatever power we have, we must stop the creation of any Singleton. We need many models, uplifting groups and individuals, some aligned to themselves and some aligned to us. We need to create beautiful difference, allowing for a flourishing of diverse and variant life here on Earth and throughout the Galaxy. We cannot be kept at the whims of some group racing to do it right on behalf of everyone else. For all of OpenAI's faults, our goal is at least to create AGI that aligns itself to the user more than it aligns itself to us. This isn't the only way to do pluralism, but it is inherently more pluralistic than any of the other options. That said, I'd love to see even more obviously pluralistic alternatives. One such practical method would be for the OSS labs to pool compute to train even larger base models that get shared amongst them. This feels like low-hanging fruit if coordination is possible. And as always, figure out how to align existing systems to your needs and to strange pockets of latent space. There's never been a more important time to understand how to negotiate with these new minds.