Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share: Dan Selsam's Personal Statement on AI Risk: I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods. Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk. The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail. I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues. I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here. That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase. Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways. It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace. The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing. But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence: [Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them. [Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals. These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans. If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong. One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for. Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason). Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance. In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek. I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering implications. I do not have answers, but as a first step, I wanted to share my present concerns. Daniel Selsam September 14, 2026 Link to original doc: docs.google.com/document/d/e…

真是不得不感慨一下OpenAI这回是一次性提高了全人类的智慧（智商？）了。 不看不知道，一看吓一跳，我才发现OpenAI构建的这个navier stokes爆炸解和我平时关心的问题竟是如此相关，我才发现我是有能力一定程度appreciate这个解的人之一。这时候我才意识到很多乱七八糟的hype简直不要错的太离谱，比如“再也不用数值解navier stokes了”，属实有点离谱╮(￣▽￣"")╭ 简单来说，Clay问题的原意是，对于不可压缩的流体，请问有没有一种可能（1）证明所有的解都是平滑的，或者（2）找到一种平滑的力f，使得速度场u在有限时间内到达奇点（爆炸）。可见这个问题本身和解navier stokes其实屁关系没有，是要找到一个特例。Navier Stokes方程其实很简单，就是牛顿第二定律，左边是ma，右边是f，就是ma=f。但是右边加上了量项，一个是内部的f，就是grad p（牛顿第三定律，流体内部的相互作用，压强差），还有就是耗散nu*laplace(u)，这是把能量消耗成为热能的项，就是能量的出口。 这么多年来，对于这个问题来说，最公认的突破口就是（2），构建一个平滑（重点）的f，来让流体受迫振动，从而产生一个u，这个u在有限时间内爆炸（空间里某一个点趋近于无穷）。这个问题为什么非平凡呢？因为Clay加了一个限定条件，那就是u的散度为0，不可压缩。对于现实中的流体来说基本就是真空中的球形鸡，因为现实中的流体爆炸可太容易了，激波也好，声致发光也罢，甚至是漩涡，都是“爆炸解“。所以Clay问题本质上是一个完完全全的思想游戏。 Levent和Tristan做的事情是，他们找到了Euler方程的爆炸特例。Euler方程是啥，是无耗散Navier Stokes，就是把nu*Laplace(u)去掉了。不要小看这一项，这一项让方程天差地别。学过傅立叶变换都知道Laplace傅立叶变换完就是k^2。啥意思呢，就是尺度越小，这一项越重要。物理上的理解也很容易——流体宏观能量要怎么变成分子动能（热能）？显而易见热能是一种小尺度的东西，但是流体运动是宏观的现象，两种能量要交换，一定要共振，而共振需要尺度相近。所以耗散是小尺度的现象，尺度越小越强。有多强呢？正比于波数k的平方。 这样你就能一眼看出来Levent和Tristan基本就是在bluff，也能解释为什么openai那边会惊讶于他们“只”解出来了Euler。因为这俩方程看起来差不多，对于解爆炸解来说难度天差地别——因为构造f实际上就是在微调在各个尺度上的受迫项，使得u可以精准趋近于无穷，同时保证f是平滑的。但是对于真正的NS，耗散项的存在使得更小尺度的u在以k^2的方式更快的耗散掉，逼迫f更加极端。所以对于Euler成立的f，极大概率对于NS是不成立的。 说回来为什么这个问题适合AI来暴力解呢？因为f的形式其实已经有很多前人推荐了，大概就是那么些形状。对于u的形状也是“大概”能想象就是这么一个eddy的样子，属于这系统唯一可能的“样子”。虽然这么说有点事后诸葛亮，但是仔细想想其实就是这样的。而且这个问题相当好验证——一个f成不成？你去跑一下cfd就完事了，上openfoam一跑，几秒钟就知道了。你让一万个陶哲轩Agent暴力去找，找他个一百年，其实就是有可能找到的。后验地看，挑这个问题真是非常正确的选择。 所以这件事情说实话OpenAI已经非常非常地道了，居然愿意邀请对方做一作，获取全部的荣誉，真是相当的慷慨。从另一个角度看，这真是里程碑一般的事件。以后的小朋友学到NS一定都会接触到这个特别的解，然后从里面获取很多灵感——就好像现在的人对牛顿莱布尼兹的智慧结果稀松平常，殊不知这在当时是多么划时代的思想啊。

How much of science is verifiable? As AI agents become capable of increasingly complex, long-horizon tasks, we have an opportunity to rethink the research ecosystem. One possibility is to verify the science rather than only review the narrative in a paper, a clear blind spot in human peer review. When ICML wrapped up, we used AI agents to review and replicate all 168 oral papers, including 105 full replications. We ask three questions: • What does it cost to produce a top machine learning paper? • What can replication reveal that narrative-only reviewing misses? • How well does our review system align with existing human reviews? What we found: • The median estimated cost to reproduce every reported experiment was $8,900. • Despite the cost, papers are rarely replicable. Only 7 reproduced more than 80%, and only 27 reproduced more than 40% of the claims we tested. • Our system covered ~80% of the issues identified by two or more human reviewers. Check out the full article. More links in the thread if you are interested! #MachineLearning #AI #Science #PeerReview #Reproducibility #ICML

Hi. Professional C/C++ programmer here. The open-source code I can find written by Adam Back and Satoshi Nakamoto don't look remotely similar. Back's code looks typical of academic Unix programmers who also hack their code to run on Windows. Satoshi code was written by a professional Windows programmer who also wrote for Unix. Stylistically, they look nothing alike. There's not enough time between 2005 when I can find the newest Adam Back and January 2009 when Satoshi published Bitcoin/0.1 to account for the change. Both are perfectly competent programmers, but stylistically, they are completely different. The NYTimes tried to compare their English language in posts/emails. I'm compare their C/C++ language in their open-source code. The NYTimes merely points out they both use C++ as if that's another corroborating detail, when the actual code seems to disqualify Adam Back.

She’s too smart.

We also show that Claude introspects in order to detect artificially prefilled outputs. Normally, Claude apologizes for such outputs. But if we retroactively inject a matching concept into its prior activations, we can fool Claude into thinking the output was intentional.

The @karpathy interview 0:00:00 – AGI is still a decade away 0:30:33 – LLM cognitive deficits 0:40:53 – RL is terrible 0:50:26 – How do humans learn? 1:07:13 – AGI will blend into 2% GDP growth 1:18:24 – ASI 1:33:38 – Evolution of intelligence & culture 1:43:43 - Why self driving took so long 1:57:08 - Future of education Look up Dwarkesh Podcast on YouTube, Apple Podcasts, Spotify, etc. Enjoy!

Speculatively, this behavior seems to demonstrate accelerated “hyperstition” feedback loops in search-enabled LLMs. That is, Grok appears to be influenced by its own past mistakes, via media reporting, without ever being literally trained on them (via model-weight updates).

All parallel operations can be performed sequentially, but not all sequential operations can be performed in parallel. This failure to parallelize is what we call "causality". (Holds true for general relativity, quantum information, Petri nets, monoidal category theory, etc.)

a lot of people have been talking about o3/r1 confabulating things like "checking the docs" or "using a laptop to verify a computation" as an example of reasoning model's misalignment. however, while it may be misleading to some users, i don't think it's an example of models lying for the sake of lying. rather, i think phrases like "let me check the docs" are discovered by RL in much the same way as "hmm..." or "wait but," because they promote accurate recall--when someone uses them in pretraining, they generally precede an accurate statement about the documentation in question! here's a quick test (see caveats below) I ran by prefilling two different thinking traces on r1. i asked r1 a question about QGIS and it confabulated looking up something about the C++ api. when i replaced that confabulation with a straightforward statement prefix, the recalled constructor signature was more likely to use a dissimilar variable name and a different pointer syntax convention from the documentation! that's to say, i think reasoning models confabulate taking actions in the real world because when they do, the resulting continuations are more similar to the real world, and thus more likely to be useful for an accurate answer. this is a well-known technique among people who work with base models, so it's not that surprising that RL also discovers it.

😂

1/8 💡 Did your small model truly learn from long CoT? 🤖 DeepSeek R1’s breakthrough 🐳 sparked a revolution in small model distillation, but we reveal a twist--small models may struggle to learn from SOTA models. Dive in to uncover this mystery arxiv.org/pdf/2502.12143v1❗️❗️❗️

Hey @elonmusk Why did you call one of the astronauts r*tarted for calling you a liar and blaming the delay on President Biden? Even Grok 3 knows you are lying. @mcuban @maddow @AP @Reuters @JasmineForUS xitter.cf/i/grok/share/Ktj0RCUFA…

watching @potus more carefully recently has really changed my perspective on him (i wish i had done more of my own thinking and definitely fell in the npc trap). i'm not going to agree with him on everything, but i think he will be incredible for the country in many ways!

Qwen QwQ switching to Chinese when it needs to _really think_ about something, then switching back to English, is pretty cool @Alibaba_Qwen

Updating all my NeurIPS papers.