Princeton CS prof and Director @PrincetonCITP. Coauthor of "AI Snake Oil" and "AI as Normal Technology". https://xitter.cf/t.co/ZwebetjZ4n Views mine.

Princeton, NJ
Joined December 2007
Arvind Narayanan retweeted
Please stop posting this take. It’s a bad take! Bankers should be concerned about cybersecurity and AI (and trust me, they are). People should not panic about their bank accounts being hacked by AI and zeroed out. If your bank account is hacked, and the money is taken, that is an unauthorized transfer under the Electronic Funds Transfer Act. As long as you provide timely notice to your bank, you’re not liable for the loss. People should be concerned about being tricked into authorizing fraudulent payments, because they often have liability in those cases. And the potential for AI to supercharge scams is definitely a concern. But if you’re going to post about those concerns, be specific and practical, not vague and ominous. Panic about the safety and soundness of the financial system serves no one.
Do people think their bank accounts are safe?
45
50
8
737
126,819
Arvind Narayanan retweeted
Firms are doing a lot to capture value from their own AI adoption, and yet often don't succeed. We are trying to understand the bottlenecks in practice better. If you have visibility into firm adoption whether as an employee, FDE, consultant, VC, or in some other capacity, we would love to hear from you. docs.google.com/forms/d/e/1F…
4
8
2
38
4,930
Arvind Narayanan retweeted
Yeah, I think it's a bit unfortunate how some of the discourse surrounding this - even from smart, knowledgeable people - has underplayed or omitted prosaic, clearly fixable factors. People often gesture at Hugging Face, etc., as self-evident vindication of worries about catastrophic misalignment leading to human extinction, but it's unclear whether the incidents we have observed are more likely under the doomer worldview than the 'normal technology' one.
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
3
5
1
50
5,460
Arvind Narayanan retweeted
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
38
95
17
814
41,321
Arvind Narayanan retweeted
AI as a Rohrschach Technology test. Ever talk to someone who was around in the very early days of the internet? They did some incredible hacking back then too. "Normal" transformative tech never feels normal.
Is it "weird" or does he maybe know what he's talking about? Anyway, for those who are open to the idea that there's a well-thought-out AI-as-normal-technology perspective on AI safety and the recent loss-of-control incidents, see my essay with @sayashk, or my interview with Ezra Klein, or the writing of many others with similar views including @joshua_saxe and @natolambert. normaltech.ai/p/the-ai-as-no…
2
2
13
2,198
Arvind Narayanan retweeted
We gave GPT-6 Astra and Claude Fable 5.1 open-ended AI research questions from unpublished papers, with a $10,000 budget each, and had the original authors grade the papers they produced. Compared to our first tests with Claude Opus 4.8 in July, we saw meaningful improvements in experimental design and resource management, but persistent limitations in creative problem solving, judgment, calibration, and communication; both papers were rejected by our experts. cruxevals.com/research-notes…
15
46
6
295
26,928
Arvind Narayanan retweeted
An update on our evaluations of AI agents conducting open-ended AI research. Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. In July we tested the ability of frontier AI agents to conduct open-ended AI research, using Claude Opus 4.8 and GPT-5.6 Sol. We gave agents research questions from two unpublished papers and thousands of dollars of API credits and compute. The authors of the original papers reviewed the AI-generated papers, following a NeurIPS-style template, and rejected both papers. Since then, OpenAI has released GPT-6 Astra and Anthropic has released Fable 5.1. Can these new models conduct open-ended research with a $10,000 budget? In follow-up experiments on two papers (one with Fable 5.1 and another with GPT-6 Astra), we observed meaningful improvements to the agents' capabilities at experimental design and resource management. But we also saw continued limitations in creative problem solving, judgment, calibration, and communication. As a result, the expert reviewers still unambiguously rejected the agents' papers. We plan to continue running these evaluations with better models, better scaffolds, and higher budgets. If you are interested in long-running agent evaluations, reach out!
31
27
5
156
10,979
Arvind Narayanan retweeted
this is totally worth a read! will shift your perspective on how warped the current direction is towards automation and how much more we can do on augmentation. on "Problem #4: How can we evaluate, and optimize for, human-AI collaboration?" I learnt a lot from our initial attempt (centaur bench: arxiv.org/abs/2608.18554), but also learn how complex/multi-dimensional the problem is. This essay does a great job of laying out some of those dimensions. I'd tried to put together something very quickly for our phd students (linkedin.com/pulse/why-measu…) and this picture still guides my thinking here.
There is widespread frustration about the AI industry’s “automate everything" agenda. But what would it take to steer AI toward augmenting human capabilities instead? In a new essay, Gabriel Unger (@_gabrielunger) and I lay out a research agenda and plan for action. normaltech.ai/p/steering-ai-… Our thesis is that the reason why AI-for-automation currently dominates isn’t just economic incentives but also deeper cultural issues in computer science as a field. And that can be changed. - We sketch four important research directions that are needed in order to build collaborative or augmentative AI. The automation paradigm largely ignores these technical challenges. - We advocate for Human-Computer Interaction and Human-Centered AI to be elevated as a subfield in status, influence, and funding, and changing AI curricula to emphasize the role of the human and decrease the centrality of the pure-automation problem framing. - We point out that there are a surprising number of efforts in economics, computer science, industry, and policy all trying to bring about a shift toward augmentation, but not enough collective awareness. Better connecting these movements would amplify their power. - We think the automation agenda is struggling on its own terms. Compiling evidence of the limits of general-purpose agents for automation and loudly making the case will encourage more companies to take collaboration more seriously as a goal. - We predict that once more workers experience this way of working and recognize that there are alternatives to the delegate-and-supervise pattern, it will generate more demand, creating a flywheel effect. - Targeted policy levers can play a small but important role in amplifying both the supply and demand for pro-worker AI. We don’t need clumsy top-down mandates. We're not against automation. But we think the goal of AI must be broader. This is the start of a long-term project. I know there are far more like-minded efforts out there than we’re aware of. I’d love to hear about some of them.
3
4
22
3,340
While automation dominates much of the AI conversation, what other research directions should we prioritize? @random_walker and @_gabrielunger argue for a broader research agenda in "Steering AI Beyond Automation." Read the essay here: normaltech.ai/p/steering-ai-…
4
1
5
1,821
Arvind Narayanan retweeted
Here's an agenda that's very attractive from the AI safety worldview and a broad coalition can get behind.
There is widespread frustration about the AI industry’s “automate everything" agenda. But what would it take to steer AI toward augmenting human capabilities instead? In a new essay, Gabriel Unger (@_gabrielunger) and I lay out a research agenda and plan for action. normaltech.ai/p/steering-ai-… Our thesis is that the reason why AI-for-automation currently dominates isn’t just economic incentives but also deeper cultural issues in computer science as a field. And that can be changed. - We sketch four important research directions that are needed in order to build collaborative or augmentative AI. The automation paradigm largely ignores these technical challenges. - We advocate for Human-Computer Interaction and Human-Centered AI to be elevated as a subfield in status, influence, and funding, and changing AI curricula to emphasize the role of the human and decrease the centrality of the pure-automation problem framing. - We point out that there are a surprising number of efforts in economics, computer science, industry, and policy all trying to bring about a shift toward augmentation, but not enough collective awareness. Better connecting these movements would amplify their power. - We think the automation agenda is struggling on its own terms. Compiling evidence of the limits of general-purpose agents for automation and loudly making the case will encourage more companies to take collaboration more seriously as a goal. - We predict that once more workers experience this way of working and recognize that there are alternatives to the delegate-and-supervise pattern, it will generate more demand, creating a flywheel effect. - Targeted policy levers can play a small but important role in amplifying both the supply and demand for pro-worker AI. We don’t need clumsy top-down mandates. We're not against automation. But we think the goal of AI must be broader. This is the start of a long-term project. I know there are far more like-minded efforts out there than we’re aware of. I’d love to hear about some of them.
1
6
4,720
Arvind Narayanan retweeted
New essay: To keep human judgment, we need AI courts. AI-generated filings are starting to overwhelm courts around the world. The most popular response I’ve heard is to fight AI with AI: If we can just give judges the right tools and training, AI will help them keep pace. But this strikes me as the worst of both worlds. Because we want human judges in the loop, we don’t get the speed and scale benefits of AI. Yet because we expect them to resolve a massive number of cases quickly, judges (or their clerks) may end up leaning on AI more than they mean to or should. This essay offers another way: a parallel AI adjudication track. An AI track — inspired by AI use in private arbitration but formally part of the court system like magistrate judges — would let courts be explicit about how human judgment is allocated. Done right, this would allow courts to process large volumes of cases while ensuring human judges devote their full attention to the cases where their judgment is needed most. Read more in @PaxMachinaMag: paxmachina.ai/to-keep-human-…
21
27
23
198
25,226
Arvind Narayanan retweeted
Do we live in a world where AIs can be wildly superhuman at every single task? Or a world where different tasks have different inherent limits, and AI’s superhuman performance at cybersecurity and math won’t translate to many consequential tasks? This is a key crux that distinguishes the normal technology worldview from the superintelligence worldview, and it has nothing to do with the pace of AI progress — it's about our different assumptions about the world. Here's an intuition pump for why I think it's the second world: different tasks have different fundamental limits. *Category 1: Some tasks are like making a building taller*: there is no ceiling to how well you can do. More compute and better algorithms can lead to wildly superhuman performance. The entire history of computing is the history of building tools that allow us to be “superhuman” at computational tasks. For example, compilers are superhuman assembly code writers, and fuzzing tools are superhuman vulnerability detectors. AI builds on a long line of such improvements: solving math problems and finding exploits in code are two recent examples. *Category 2: Some tasks are like making a ball rounder*: there is only so much room for improvement, and the best AI systems can’t wildly exceed humans. Take persuasion. Some people believe AI can persuade anyone of anything, even convincing military commanders to launch nuclear weapons. In fact, “superpersuasion” is one of the pathways through which many catastrophic risk scenarios unfold. But the AI as Normal Technology thesis is that there are many consequential tasks that are limited by external factors, so such wildly superhuman performance is not possible. This is especially true against humans who use simple computational tools. Tasks like persuasion, or predicting the outcomes of social events (such as elections), fall into this category. Other tasks, like writing, can’t be ranked on a single scale; different people find different writing styles compelling. AI writing also quickly wears thin: even if a user is impressed by AI writing at first glance, after reading thousands of words of the same type of prose, it becomes mundane and irritating. All of these are different from Category 1 tasks like playing chess, where even expert humans can’t do much better than deferring to an AI system at each turn. *Category 3: Some tasks require experiments in the real world*, and the bottleneck isn’t the computational task; it’s the data collection. One example is Moderna’s Covid vaccine: it took two days to design the vaccine, but almost a year to test and authorize, even during a record-breaking vaccine-development effort! This is why a “superintelligence” won’t cure cancer from inside a datacenter: the bottleneck to many of AI’s most sought-after real-world impacts (such as developing cures to diseases) is in the real world, and requires experiments that we won’t (and shouldn’t) hand over to AI. People concerned about AI risks often assume that we live in a world where almost everything fits into Category 1, and we might soon see AIs with wildly superhuman performance at all tasks. I agree it would be scary to live in that world. One reason @random_walker and I are more optimistic about our ability to remain in control is that we don’t think we live in the Category 1 world.
71
121
35
659
78,616
Arvind Narayanan retweeted
If AI labs want personal assistants to take off at scale, they need to stop marketing vacations, reservations, party planning, etc. These are not problems. Focus on *the* problem facing Americans: the cost of living. Agents can offer a ton of help. Some ideas: (🧵)
165
186
121
2,444
402,638
Arvind Narayanan retweeted
Fascinating reading about how we used to write about malware and viruses back in 1999. In a decade I suspect we'll look back at how we talk about AI safety today and it will feel similar. schneier.com/crypto-gram/arc…
11
43
6
241
14,595
Arvind Narayanan retweeted
Joshua Saxe
@joshua_saxeOct 7
Reaction to the @slatestarcodex vs. @sapinker AI x-risk debate The main problem with this debate is that both interlocutors are mainly focused on AI as a disembodied technological object that's fallen down from heaven, not a congealed mass of human decisions. Scott has understood that there's nothing ontological about deep learning that'd prevent us from killing ourselves with it and that this risk gets worse as AI scales. But this is a trivial observation! Here's how to kill everyone with AI: put Opus 8.1-Killeveryone in charge of 100 bio labs; skimp on safety post-training; connect it to the Internet; let it develop affective relationships with the lab post-docs and its nihilist friends on Reddit; don't monitor it; have it work as a swarm to solve 'BioKillEveryoneGym' over a holiday break... In fairness, Pinker and Alexander do touch on social processes. But if one is concerned with scenarios like these, 90% of the debate should focus on the social processes by which we risk creating them - incentives, culture, guardrails, ... - and on why we won't adapt and reach a reasonable equilibrium somewhere along the way. To be clear, I'm very concerned about AI harms - including tail harms - but not very concerned about extinction risks. The burden is on Pinker and deep learning denialists to show that AI couldn't kill everyone under any circumstances (hard to defend); but the burden is on Alexander to show the social mechanisms by which we’d make the long sequence of low-probability unforced errors that would actually allow it to do this (seems hard to defend). We could kill a lot of people with nuclear power plants. Here's how: don't monitor them. Fire all their safety engineers. Build them in the middle of all of the world's population centers and then cut off their water supplies whenever city budgets run low. That we haven't done this is heartening; we should understand why. That there've been many accidents with dangerous technologies, especially in the earlier phases as we thrash towards a safety equilibrium, is disheartening; we should understand why those happened, too, and apply all those lessons to AI.
Scott Alexander has written an open letter in response to Steven Pinker.
25
28
8
130
16,170
Arvind Narayanan retweeted
🚨SciConBench accepted at #NeurIPS2026! Paper: arxiv.org/abs/2606.11337 Can AI *synthesize* scientific conclusions in health? We find that, on medical conclusions, we are far from it! Check out our live dashboard: sciconbench.cs.princeton.edu… w/@hayounggjung and @korolova
3
18
2
57
5,344
Arvind Narayanan retweeted
Can two superforecasters + machine outperform machine in forecasting? When the machine first beat a human in chess there was a point in time when man + machine > machine. At Preseen we’ve built out one of the best AI systems for forecasting. We wanted to see if two superforecasters working with our system could outperform the system itself. They ended up narrowly beating the system, but what we learned was fairly surprising. A few takeaways: - Being a domain expert doesn’t map onto using an AI system well. Guiding superhuman (or at least on-par) AI requires a different skillset. Effectively working with an AI system requires an additional skill: knowing when and how your own expertise can improve its output. - Evaluating a centaur approach is non-stationary since the human’s behavior with the agent changes as they learn more about the constraints of the agents. If an agent has a few dimensions a human can interact with, the human’s use of those dimensions can change dramatically. - Human agreement could reinforce a mistake. On the three questions where overrides cost the most, the humans moved toward the community prediction. What looked like a sensible correction supported by other forecasters ended up hurting performance.
6
5
28
4,326
Arvind Narayanan retweeted
Y2K went down in history as a public overreaction. Apparently, that’s incorrect? Programmers and critical infrastructure people worked for over a decade to prevent problems. A lot of them never publicized it when they fixed things in their own orgs. (Each fix isn't headline news.) Time magazine says a lot of people stockpiled supplies "because they didn't have a sense that programmers were on the job." A few things did go wrong. There was an issue at a couple of nuclear plants. A video rental store charged a customer a century of late fees. But overall, everything worked out fine. Skeptics took that as proof nothing was ever going to happen. The people raising alarms weren't exactly proven right! Because years of hard work meant nothing happened. The national and local coordination was so good that President Clinton's Y2K czar was comfortable getting on a flight at exactly midnight. The only people on that flight were him and reporters. Another example of how saving the world and preventing disaster gets remembered as wasted money because nothing happens.
5
4
2
30
2,650
Arvind Narayanan retweeted
Have been noodling on an actual piece about this, but I think this sentiment somewhat misunderstands the Carnegies/Rockefellers and how their foundations actually came about/became effective. Carnegie had been giving for decades, but he really kicked it into high gear after he sold Carnegie Steel and retired. He was 65. Likewise, even though Rockefeller began giving much earlier, the Rockefeller Foundation, his philanthropic capstone, wasn’t chartered until 1913…when Rockefeller was 73. A lot of today’s fortunes are 1) still being made and 2) being accrued by people who are much earlier in both their careers and their philanthropic lives than Carnegie and Rockefeller were when their philanthropic institutions reached maturity (not that this should absolve them of any sense of noblesse oblige *today*). These great philanthropies also can’t be separated from the era in which they were created: Protestant ideas about stewardship, the general Progressive Era reform culture, anxieties about (and outright hostility toward) dynastic wealth, reputational pressures on the new industrial rich, and limited U.S. state capacity wrt education, public health, and scientific research, were all contributing factors to their inception. The second and more important piece of the story to me, though, is that sentiments like this give too much credit to the billionaires themselves. The *other* thing each of these men had was brilliant advisors. Rockefeller, for example, did not wake up one day and independently derive the idea to found one of the country’s greatest research universities (UChicago). It actually emerged from a network of Baptist educators and organizers, among them Frederick Taylor Gates, who would become Rockefeller’s principal philanthropic adviser. There are so many examples of this throughout history, from Walter Damrosch (the young conductor who pitched the idea for a great concert hall to Andrew Carnegie, and you know the rest) to George Grey Barnard, the sculptor and art collector whose collection formed the basis of the Met Cloisters (currently working on assembling what I hope will be a comprehensive list of such efforts). These were people capable either of converting a benefactor’s general desire to do something important into a specific institution, field, collection, research program, or public good, or simply of presenting them with an extraordinary idea worth backing. Which is all to say that patronage is often a team sport! Yes, the capital ultimately has to come from somewhere, but I really do believe now might be the best time in history to become the person a very wealthy person relies upon to understand a particular domain, or to create the kind of cultural institution that you believe is important. So yes, one interesting question is, “Where are the Carnegies?” But also…where are the Frederick Taylor Gateses?
It’s incredible how much wealth SF entrepreneurs have created and how little of it has gone toward building culture. Where are the great museums, new architecture, music venues, public spaces? Silicon Valley should aspire to leave behind more than ephemeral software.
19
33
3
334
32,756
Arvind Narayanan retweeted
Does anyone disagree that pretty much every piece of empirical evidence we have about actually existing AI points toward a labor-augmenting technology with a more-or-less ordinary risk profile (no worse than cars, guns, nuclear power, social media, etc.)? On the labor front, the evidence of minimal / inconsequential employment effects so far is undeniably strong (see: aleximas.substack.com/p/has-…). On the risk front, the recent spate of "rogue agent" incidents was akin to a set of natural experiment in which much of what could go wrong did, and the costs were comparable to or less than your typical high-profile cybersecurity breach (several of which happen around the world every month). Not only that, but the base rate of AI-related hacking incidents is extremely low, and with rare exception, model behavior during real-world use cases is exceedingly well-aligned. Plus, as Anthropic recently argued (anthropic.com/research/glm-5…), the open-source GLM 5.3 is around as capable as Claude Mythos at cyber-exploitation while also having much lower safeguards, meaning that every additional day absent some dire cybersecurity incident is further evidence against catastrophic AI cyber-risk. Taken together, these facts imply two constraints on arguments in favor of catastrophic AI risk: 1) those arguments must either pass through future, rather than current capabilities, or future, rather than current varieties of AI adoption or implementation (or they must claim we have been unrealistically lucky); 2) those arguments must also must pass through some specific capability or adoption threshold, rather than rapid improvement or adoption as such, since dramatic capability improvement and rapid adoption have already occurred alongside demonstrably low risk of catastrophic harm. It is therefore natural that those concerned about catastrophic risk commonly appeal to recursive self-improvement, or RSI in order to make the case that things will get much scarier very soon. But as @ramez has convincingly argued (noahpinion.blog/p/wheres-the…), existing data on the reality of RSI within frontier labs neither suggests an inbound trend-break, nor the accelerating returns that might produce one. On the contrary, as is unsurprising to those familiar with the generally log-linear relationship between compute and model performance, all available signs points toward diminishing returns. And this is before considering external factors that could slow capability progress further, including low-hanging fruit effects related to running up against more complex or tacit-knowledge-based tasks for which performance data and verifiable reward signals are scarcer, plausibly incoming financing constraints related to the industry's massive capex commitments (see: econjared.substack.com/p/a-b…), growing political backlash, both to data centers in particular, but also rapid AI development writ large, and new regulatory bodies and industry reporting frameworks (see e.g. markey.senate.gov/news/press… and axios.com/2026/08/11/open-so… ), which will require labs to proceed with greater caution from here on out. I am therefore led to conclude both that catastrophic AI risk is likely to remain negligible for the foreseeable future and that the only counterarguments concern massive black-swan-style capability uplifts no existing data imply are on the horizon, and/or particularly irresponsible or dangerous forms of AI adoption that there are also few precedents for so far (and that we are actively developing norms, regulation, and security measures against). Neither is impossible, meaning we should remain vigilant and that AI safety efforts will continue to be well worth it in the coming years. But I see no way around the conclusion that something akin to the "normal technology" view is superior on the empirical and argumentative merits to whatever variety of doomerism. What am I missing?
50
34
12
255
73,420