I’ve read this a couple times now, and while there is, frankly, a lot of sloppy argumentation (which I will get to), the charitable, simplified read of Dwarkesh's position seems to be:
1) AI needs major breakthroughs to achieve AGI/ASI (continual learning, sample efficiency, etc.) — this is a view he’s held for some time now, in line with Ilya
2) These breakthroughs, and the ML research skills required to obtain them are verifiable, and thus current, pre-breakthrough AIs can hill-climb on these sorts of problems and capabilities to solve them (like we saw in math)
3) Once these breakthroughs occur, we *may* see rapid, recursively compounding gains to intelligence, leading to ASI in a very short period of time
There's a ton of other stuff in the essay, but I think this is the basic crux. Christiano basically articulated this same argument in his Dwarkesh appearance, and if you look at AI-2027, it essentially follows the same trajectory: Agent 2/3 makes a ton of algorithmic progress, allowing Agent 4/5 to explode.
In general, I think the high-level narrative here is...fine? It’s certainly one plausible outcome. But, imo, it is a much narrower path than I think many are giving it credit for—a lot of things actually have to go just right for this to play out. And here's where I return to the other, messier arguments, because many of the intuitions and arguments driving Dwarkesh's view in favor of near-term ASI (e.g., EO2027) as a highly probability outcome (~50%) are, imo, quite bad:
- “[We'd see] a jump as big as GPT-3.5 to GPT-6 Astra in just 12 months” -> No, under this scenario we’ll see ~4 years of AI progress in 1 year. If there are diminishing returns, then this won’t be as extreme as the jump we saw from GPT-3 to GPT-6. It very well could be, but that would require innovation to overcome the diminishing returns we'd expect in the base case.
- “The other big piece of evidence for transfer is human evolution” -> Except if AI was following the sort of pattern we saw for human evolution, we should have seen better transfer long before AI started solving millennium problems. This spikiness has been one of the big surprises of AI!
- “Rather, my claim is that scaling up RL training will give AIs the abilities necessary to ramp quickly in new contexts…it’ll go through an evolutionary process that involves centuries of corporate competition to figure out the optimal organizational techniques.” -> These are in direct conflict! If ramping requires RL runs that simulate centuries of competition within “digital economy full of millions of other AIs” then that’s quite bearish for ramp speed (relatively) given the compute and wall-clock requirements!
- “I have a lot of 20-something-year-old friends whom I’ve watched accomplish incredible things and build world-leading companies and ventures in domains they entered into with no previous experience.” -> lol. rip leopold. But also, again, these are humans, with human sample efficiency. If AI gets there, sure. But how soon it gets there is the big crux.
- “Obviously, some pieces of feedback—about how your actions impacted the quarterly earnings, say—require months of serial time. But even granting that, we should expect these future AIs to be able to get to top human performance in almost all domains within months of being deployed.” -> …no that’s not how that works at all because one set of earnings isn't enough to robustly learn from and you can’t re-run the same real-world conditions.
- “[Sample efficiency & online learning are] AI’s equivalent of a Millennium Prize problem. And mathematics has already shown us what happens to such problems” -> Sure, but there’s a difference between “AI can solve a Millennium-level problem” and “AI can solve *this specific Millennium-level problem*” -- if the blocker is a specific problem, the timing spread becomes much wider.
- “My answer to this [experiment/wall clock/compute] objection is that each agent will still have access to a ton of compute. Frontier labs have been more than quadrupling their compute every single year.” -> But that’s what they’ve needed to do keep up with their current pace! Even if the labs maintain their current pace of compute growth (and like, will they even? capital markets really do not seem accommodating and Dwarkesh's roommate just lowered his compute projections), we'd still need a step change in pace above that!
And that's not even covering the biggest crux of all: whether the returns to intelligence continue to recursively compound faster than diminishing returns eat into them.
Now, are any of these objections fatal? No. Do they “disprove” hard RSI/an intelligence explosion? No, ofc not. Am I kind of nitpicking here? Yeah, 100%.
And to Dwarkesh's credit, he does anticipate and acknowledge a scenario (the “spark”) where these objections play out.
But the reason I've bothered to rant this long is that I see a lot of AI/RSI discourse follow this pattern: where 50% of the arguments are reasonable and grounded and appropriately hedged, and then the other 50% are these sorts of sloppy intuition pumps and what-ifs that include a ton of detail but a noticeable lack of rigor. And I can’t help but get the sense that it is the latter 50% that is really driving people’s predictions and expectations here, as it leads to sentences like this:
“In the spark scenario…it takes years to get to the same endpoint that the software-only singularity would have achieved in months: a superintelligence that is to us what we are to chimps.”
Note that Dwarkesh thinks these two outcomes (spark vs. software-only singularity) are equally likely. 50/50. I’m sorry, but that’s absolutely nuts. Even Christiano put the likelihood of ASI by in 2027 *absent any slowing down* at only 1 in 3, and I still think that's too bullish.
Like, yes, I fully anticipate that we will have automated AI researchers (or something close to them) in 2027, and the universe where we go from that to a software-only singularity is not entirely out of the question. But obtaining that universe relies on a confluence of various frictions, diminishing returns, and wall-clocks being overcome, a somewhat specific set of algorithmic innovations being made or obviated, and an assortment of other factors playing out just-so to enable that perfect storm. Even a fairly rapid exponential doesn't qualify when you're talking about full-blown ASI in 13 months. It may not be impossible, but it's certainly not a 50/50 outcome.
konrad retweeted
I don't know if you actually need legislation to get real commitments out of these companies. But you definitely need the threat of legislation. You need them to believe that the alternative to doing this is MORE transparency and oversight.
konrad retweeted
New EA cause area: flood the Bay Area with babies
Works in Progress' @RuxandraTeslo says Silicon Valley's willingness to hand humanity's future over to AI may stem from how few people there have children:
"A lot of people think about this only from the perspective of their own lives, not about the future."
konrad retweeted
Introducing Claude Psychiatry: add Claude to your group’s signal chat or discord and it builds detailed psychological profiles, diagnoses, and dispenses medication via select partners while you’re still talking.
Starting to roll out in beta on Team and Enterprise plans.
konrad retweeted
Replying to @politicalmath
my mute list probably has thousands of entries at this point
i credit my enjoyment of this platform to muting freely
konrad retweeted
Was funny to watch everyone in Tech finally, collectively, realize tech's PR was awful and that the general public was gonna eat Tech alive in short order if it wasn't helped.
So, days later, every big Tech figure on X tweets simultaneously "It's okay for the powerful tech CEO to have affairs and do whatever he wants, accountability is corruption"
Like, yes, of course, Jensen leaving Nvidia is bad, BUT you people are really really not helping the broader issue facing the entire ecosystem.
It's almost like tech subconsciously wants to be crucified
can estrogen save jensen...
NVIDIA announces CEO transition. Read the release: nvidianews.nvidia.com/news/n…
konrad retweeted
Characteristically great Kelsey reply. But on this:
“I agree it would be cool if there were an organization focused on effective institutional capacity building in poor countries, but the reason I and man others don't donate to this is not that everyone fails to realize how important institutions are, it's that even though many people have worked very hard on the problem for a long time they do not have a way better than the nets to turn my money into GDP growth in poor countries.”
May I introduce our Global Growth Fund led by @JustinSandefur? coefficientgiving.org/funds/…
Before we launched this I had some of Kelsey’s skepticism about tractability here. This thread talks about how I changed my mind: teapawt.org/albrgr/status/18466225…
Couple thoughts:
1) After several people pointed out that Lyman couldn't substitute all-cause mortality for malaria-specific mortality like he had done, he fixed it. His initial estimate for $/lives saved was $19000; revised it’s about $8,200.
2) I think, though I could be wrong about this, that his calculation to get $8200 still has a problem: he calculates the costs of the distribution based on the whole population but the benefits based on 75% of the population (since he argues the other 25% is already sleeping under nets). But as he himself identifies, the nice thing about an intention-to-treat analysis is that you don't need to do this and shouldn't: you just ask how much to expect mortality to fall in a village after handing out the nets, and the answer is 17%. (You only care about the fact some people are already sleeping under nets if that makes your village unlike the one where the 17% was measured; more on that below.) If he changed this one factor this would make his numbers quite close to GiveWell's.
3) He then applies an adjustment for the long-term: "If bednets reduced short-run all-cause mortality by 18% in the 1990s and maybe reduce it by (who knows?) 14% today, they probably reduce actual child mortality (i.e. across 5 years of early childhood) by 7-11%."
But GiveWell actually accounts for that even more stringently by just excluding all benefits past the first two years, one of several places where they are intentionally conservative.
4) He accuses GiveWell of misrepresenting Cochrane's statistics but they actually just report a bunch of different numbers and I think he mixed them up. He complains "GiveWell cites that review to support a “45% reduction in malaria incidence.” That’s an error, that review did not find a 45% reduction in malaria incidence." But the review absolutely does report almost exactly that: "Insecticide-treated nets also reduce the incidence of uncomplicated episodes of Plasmodium falciparum malaria by almost a half (rate ratio 0.55, 95% CI 0.48 to 0.64; 5 trials, 35,551 participants, high-certainty evidence". Rate ratio 0.55 means a 45% reduction.
5) He expresses concern that a lot of the Cochrane studies are old and describe a different environment (Africa in the late 1990s/early 2000s). However, GiveWell's most recent grant to nets in the DRC - which I believe that my donations helped to fund - is based on data from a 2023-2024 large scale demographic survey in the DRC, so the nets are being distributed in an environment very close to the one studied. It found about a 14% all-cause mortality reduction in the first year after a distribution, close to the 17% in Cochrane.
He also expresses concern that Cochrane's mortality estimate here is low-powered but I don't think it is (the analysis says "rate ratio 0.83, 95% CI 0.77 to 0.89; 5 trials, 200,833 participants, high-certainty evidence") so it's between 11% and 23% with a 95% CI!
6) He also raises questions about whether there are more people trying to hand out nets than nets to be handed out. GiveWell looked into that and determined that in the DRC, there was not enough funding to do all the distributions that were indicated, in part because the Global Fund caps how much it'll spend in any one country.
7) Finally, he argues we should instead - from an altruistic perspective - spend our money however maximizes GDP. But I think there's a serious case that health interventions like combatting malaria are necessary for GDP growth to take off in DRC, just like the American South was kind of a backwater until we eradicated malaria there. Being constantly sick is horrible for productivity. Being only able to hire employees who are constantly sick is horrible for a business environment. GiveWell estimates that a $1,000,000 donation increases adult income for 80,000 children by 0.6%, which means that much of their modelled benefit of malaria nets *is* their GDP effects.
My best guess is that if you were a hard-nosed GDP maxxer you'd invest a lot in reducing the disease burden in poor countries. Or maybe you'd go all in on making AI happen as fast as possible, something I rarely see EAs criticized for failing to do. I would probably donate more money to making AI go well if I had more of an idea of how to do that and it wouldn't create a bunch of conflicts of interest with my reporting work.
I agree it would be cool if there were an organization focused on effective institutional capacity building in poor countries, but the reason I and man others don't donate to this is not that everyone fails to realize how important institutions are, it's that even though many people have worked very hard on the problem for a long time they do not have a way better than the nets to turn my money into GDP growth in poor countries.
There was a bit of a Our World In Data article I read this fall that has stuck with me:
"I’ve never had malaria, but I know several people who have. They describe it as one of the worst experiences of their life.
What starts out feeling like the flu — which is bad enough — can quickly progress into something far worse. The body swings from intense fever and sweats to chills deep enough to make you shake uncontrollably. Everything aches, from your muscles and joints to the pounding headache that makes it impossible to think clearly. Your body is so fatigued that it’s a battle to make it to the bathroom when the nausea and vomiting arrive.
That’s what a pretty standard infection feels like. For some, it goes much further. Here is how one account describes it:
“I awoke to what felt like lightning going through my legs, and then spreading through my body and in my head. Probably the worst headache, body aches, and chills you could possibly imagine. It felt like I was being stung repeatedly by an electric shock gun and could barely control my movements. The pain was so intense; I actually believed I was dying, literally crying out in pain so bad that I was taken to a 24-hour clinic that night at 3am.”
In Senegal, the median child experienced 43 attacks of malaria over the course of their childhood. My hard-nosed productivity obsessed persona says "I bet that's rough on their human capital development, and it's quite cheap to prevent." But mostly I don't, actually, say that to myself. I just buy some nets. If something even better comes along I'll do that, but I haven't seen it yet.
konrad retweeted
I basically feel pretty confident at this point that we're barreling toward something at least akin to the Industrial Revolution and a lot of public intellectuals are forming their opinions about it mostly via confused vague popular slogans that emerge like a game of telephone
konrad retweeted
Replying to @BethMayBarnes
Similar to the point put beautifully here
I encourage those at frontier AI companies to read this quote, from Ellsberg to Kissinger back in 1968:
“Henry, there’s something I would like to tell you, for what it’s worth, something I wish I had been told years ago. You’ve been a consultant for a long time, and you’ve dealt a great deal with top secret information. But you’re about to receive a whole slew of special clearances, maybe fifteen or twenty of them, that are higher than top secret."
“I’ve had a number of these myself, and I’ve known other people who have just acquired them, and I have a pretty good sense of what the effects of receiving these clearances are on a person who didn’t previously know they even existed. And the effects of reading the information that they will make available to you."
“First, you’ll be exhilarated by some of this new information, and by having it all — so much! incredible! — suddenly available to you. But second, almost as fast, you will feel like a fool for having studied, written, talked about these subjects, criticized and analyzed decisions made by presidents for years without having known of the existence of all this information, which presidents and others had and you didn’t, and which must have influenced their decisions in ways you couldn’t even guess. In particular, you’ll feel foolish for having literally rubbed shoulders for over a decade with some officials and consultants who did have access to all this information you didn’t know about and didn’t know they had, and you’ll be stunned that they kept that secret from you so well."
“You will feel like a fool, and that will last for about two weeks. Then, after you’ve started reading all this daily intelligence input and become used to using what amounts to whole libraries of hidden information, which is much more closely held than mere top secret data, you will forget there ever was a time when you didn’t have it, and you’ll be aware only of the fact that you have it now and most others don’t….and that all those other people are fools."
“Over a longer period of time — not too long, but a matter of two or three years — you’ll eventually become aware of the limitations of this information. There is a great deal that it doesn’t tell you, it’s often inaccurate, and it can lead you astray just as much as the New York Times can. But that takes a while to learn."
“In the meantime it will have become very hard for you to learn from anybody who doesn’t have these clearances. Because you’ll be thinking as you listen to them: ‘What would this man be telling me if he knew what I know? Would he be giving me the same advice, or would it totally change his predictions and recommendations?’ And that mental exercise is so torturous that after a while you give it up and just stop listening. I’ve seen this with my superiors, my colleagues….and with myself."
“You will deal with a person who doesn’t have those clearances only from the point of view of what you want him to believe and what impression you want him to go away with, since you’ll have to lie carefully to him about what you know. In effect, you will have to manipulate him. You’ll give up trying to assess what he has to say. The danger is, you’ll become something like a moron. You’ll become incapable of learning from most people in the world, no matter how much experience they may have in their particular areas that may be much greater than yours.”
"….Kissinger hadn’t interrupted this long warning. As I’ve said, he could be a good listener, and he listened soberly. He seemed to understand that it was heartfelt, and he didn’t take it as patronizing, as I’d feared. But I knew it was too soon for him to appreciate fully what I was saying. He didn’t have the clearances yet."
konrad retweeted
Sometimes I spend hours thinking about politics and stewing in frustration over the cynicism everyone seems to blindly hold onto. The fundamental story of civilization is one where goodness and cooperation has repeatedly triumphed, and the present is no different.
konrad retweeted
By default, you should expect an AI-driven industrial expansion to concentrate in the US, just as the data center buildout has.
Building fully automated factories requires the same things as data centers: lots of private land, easy permitting, and abundant energy. For the data center buildout, only a few regions in the US (e.g., West Texas) have been able to provide those things.
I find this somewhat worrying, and it does raise the importance of getting SEZs in non-US democracies (though doing SEZs well is really hard).
konrad retweeted
... with nicer formatting: lesswrong.com/posts/oL7aFoRm…
konrad retweeted
If just a few lab staff took on verification work full-time, verifying a US-China pacing deal could get much more feasible within months.
Here are 8 specific things lab staff could start on today to help make international verification possible: 🧵
konrad retweeted
Thanks for the response @ben_golub! Speaking for myself, I actually judge the output of the efforts mostly on if they engage with interesting and important questions, and engage with the magnitude of the change - which mostly means using assumptions which are not in the category "this is obviously not going to be true". As an added bonus, the questions can be decision relevant.
As @alexolegimas wrote in a different response, and I agree with, I want and I'm happy to argue we need models which are structurally innovative (this is what I tried to gesture at with AI not being exactly like capital or labour so far, and not being a normal technology).
Maybe it's easier to show using examples - here are some questions I currently think about
1. Should we speed up or slow down emergence of "automated bundles of capital and AI" (ie sovereign AI, AIs without principals)? What are the macro implications?
2. To what extent everything, including various good civilizational equilibria, works because of bounded rationality (where humans are within some not too wide range) and frictions?
3. What can we say about equilibria between agents with extremely different profiles of bounded rationality?
4. What are the elasticities and frictions around RSI?
5. How to protect the processes leading to formation of people's preferences?
I would love to read or skim an "AGI-pilled" paper every few days engaging with any of these. Who is working on that?
Also as a side-note - I don't want people to agree with me; actually don't know what agreeing with me on answers would mean - I don't know the answers, or have huge uncertainty about them. I would love people agreeing with me the questions are important and urgent.
As far your answer is close to "You don't understand: the field of econ is structurally not oriented toward answering the most important econ questions now. Most people work and will work on modest, backward-looking empirics, or make modest change assumptions" my response is "Perhaps the field can and should raise to the occasion? Perhaps more people can jump sub-disciplines and work on forward-looking policy-relevant quetions? Who else should orient the field toward what's important rather than people inside?".
Ultimately if almost all economists will stick to their sub-disciplines and "post-AGI as if nothing happens" assumptions justifiable to referees... who will answer the questions?
konrad retweeted
Consider that the people who were right about AI-enabled cyberattacks might be right about AI-enabled bioweapons
konrad retweeted
I'm only 2 minutes in and there have already been like 5 gems





































