I thought I'd try to reconstruct Bryan's discussion with a 13-year-old about the minimum wage based on the below tweet: (@bryan_caplan let me know if anything is inaccurate)

All AI related incidents that have happened so far have tiny costs compared to benefits that have already materialized, let alone will materialize in the future. Net benefits will hinge on whether or not we get catastrophes (loss of control / biodisaster / wars)

This view of the future seems fairly naive to me. I don't think the reasoning of a superintelligence would be "legible to us" in any meaningful way. Even among humans, which exhibit a very narrow range of intelligence, core normative concepts like efficiency are legible to only a tiny fraction of the population. A superintelligence could give us a dumbed down explanation for why it believes a given course of action is best, but the idea that humans will need to vote to ratify this course of action is somewhat ludicrous, and an overextrapolation from human norms where there are strong pragmatic reasons to prefer democracy to alternative institutions like meritocratic authoritarianism. I think it's correct that there is intrinsic value to democratic decision-making even if it leads to worse outcomes. But this intrinsic value is not infinite and is not particularly relevant in a world where some entities are far better at normative reasoning and orders of magnitude more efficient at achieving goals if not bottlenecked by dumber and more fallible human review.

“That sounds like sci-fi,” he said of computers programming smarter computers, in a world of conversational robots, driverless taxis, personalized gene editing, and brain implants that turn thoughts into speech.

Current frontier models seem to nearly always err on the side of massively overfitting models if you ask for their advice. If some coefficient varies by age, they'll immediately suggest replacing 1 parameter with 11. They have no sense of the value of parsimony.

I wonder if we will see more AI-polarization by age as both the benefits and risk become more clear. The net benefits of moving fast seem considerably larger for the elderly, who have more to gain from medical benefits and face less labor market risk.

Even if humans maintain control of AI and labor markets exist, we will have to get used to the fact that in nearly every analytical domain, AI will be doing things that humans understand at only a very high-level or not at all. That's how comparative advantage works.

When you debate AI consciousness or questions like the moral status of animals, it's always worth keeping in mind questions like: 1) How would the Greeks have discussed how many pieces an apple can be cut into? 2) How would the Romans have discussed the distance to the sun and stars? 3) How would Tang Dynasty physicians have explained why people are ill? This is not meant to be dismissive of philosophical questions! The point is not that the Greeks, Romans and Tang Chinese would have gotten these questions wrong. Ptolemy estimated the sun was 4.8 million miles away. That's pretty impressive! (he was only off by a factor of 20 or so). He wasn't perfect either -- he was off by 5-6 orders of magnitude for other stars. The lessons from these examples are: a) Some answers are much more reasonable than others. You'd do a lot better to trust Ptolemy than to trust Gallus who thought the stars were 50,000 miles from earth. b) These people weren't stupid, but they lacked conceptual frameworks which took centuries to develop which were necessary to understand what a reliable answer would even look like. For some formulations of this question, we still don't know how many pieces an apple can be cut into without a complete theory of quantum gravity. c) When a field lacks reliable methods for discovering the truth, we should be deeply suspicious of people who claim confident understanding. Even if they are right on some points, they likely have deep and fundamental misunderstandings about the nature of the phenomena in question. People without reliable methods of discovering truth should be uncertain not only about what is true, but about the conceptual frameworks they use to formulate their conclusions.

Just as mathematicians are now learning how many insights lie in the "convex hull" of existing ideas, the rest of science is about to learn this lesson at even more dramatic scale. Few realize the ridiculous inefficiency of the pipeline from theoretical innovation to practical use. Early versions of GPT made it easier to find existing knowledge. But finding a theorem is not the same as recognizing that it offers a radically different way to conceptualize a problem. Doing the latter requires expertise across fields that almost nobody possesses. Most of the people doing applied work are completely unaware of the conceptual tools that are being developed in more theoretical fields, and the people doing high-level theory do not understand the practical problems applied researchers are trying to solve. Almost no one doing practical work of any importance has an operational understanding of the web of ideas underlying the langlands conjecture or condensed mathematics, and virtually no one doing biology, chemistry or economics is knows anything about 90% of the math used in theoretical physics. And for good reason! Most of these ideas would be completely useless. But the scope for possible innovation from drawing the rare useful connections is immense. Think of Paul Samuelson borrowing ideas from thermodynamics to model equilibrium in economics, or Ed Witten using quantum field theory to revolutionize geometry. A few startling successes, not feasible to most: trying to venture into a more theoretical or more practical discipline to find insights of use is like searching for a needle in a haystack. What I suspect we will see in the next few years is a scientific revolution brought on by models that combine deep theoretical insight with deep practical knowledge and judgment. Existing publicly available models are lacking in all three departments, but getting better. When searching for useful connections between distant fields becomes cheap, even a low hit rate could produce an explosion of innovation. Even without new ideas (which will also be forthcoming...), the convex hull of existing science would push far beyond the status quo.

The idea that math is verifiable but other tasks are not is misleading. "Verification" is a proxy for what really matters: access to a cheap signal that is sufficiently correlated with the ground truth. Better theories turn expensive outcome measurements into dirt-cheap predictions. We are not at the level of God-like theory-building yet, but there is no ceiling in sight. In plumbing, we can check if the pipe leaks. With pharmaceuticals, we can check in an RCT if a patient lives longer. The problem is, doing these things requires time and money. They require time and money because of the current state of analytical modeling. A good plumber doesn't need to wait 6 months to know if they did a good job, because they know that features observable *right now* strongly predict whether the pipe will continue to leak in 6 months. But a good plumber still needs that experience, right? The need to collect the data to see the 6 month correlates of actions you take today reflects a shortcoming of models: with an accurate enough model of the physical world, a plumber could infer that a pipe won't leak in 6 months without ever having plumbed before. If you're an engineer, you might respond, "Ha! Ridiculous! Something always goes wrong that you can't anticipate." What if the pipe happens to be proximate to a freezer, making it unusually cold. You would never realize this without testing. That is true for humans because our theories of the world often miss things because we have limited memories and limited ability to think through the consequences of stuff we already know. But a still smarter plumber would infer from their theory that they need to check for this type of issue. You still might not be able to predict a meteor strike, but superhuman plumbing ability doesn't require perfect prediction -- it requires models good enough that they can beat someone with experience. We already have robots that can one or two-shot simple physical tasks they have never tried before, and perform passably at more complex tasks (tech.yahoo.com/ai/chatgpt/ar…). This capability will only increase as models improve. The idea that models won't be superhuman at real-world tasks because these tasks are expensive to verify is not coherent. Nothing is intrinsically expensive to verify. Verification is conditional on a theory, and the ability to formulate good theories is accelerating rapidly.

The most consequential fallacy in the world today may be: "nerds can't use math to become good at football or popular, so computers can't either"