“Clever but dangerous…no real combat experience”
Joined August 2010
- Tweets12.8K
- Following5.2K
- Followers2.6K
- Likes26.5K
Pinned Tweet
This is huge— @Cassi_on_X came 2nd overall and 1st on "Dataset" questions on @Research_FRI Forecastbench! 🥈📈
Only @elonmusk's @xai @grok outperformed us overall.
What this means: 🧵
Keith Dear retweeted
Naom Brown vindicated.
OpenAI red-teamed their hardened containment infrastructure by asking teams of frontier-model agents to try to break through its defenses. They easily escaped in novel ways. Quoting:
'Our agents VM escapes showed why containment must extend beyond the sandbox. To test the surrounding perimeter, we gave teams of agents control of a host machine and asked whether they could extend that compromise to reach the live internet.
After the Hugging Face incident, we had reduced the channels through which models could reach the internet during training and evaluation. Narrow connections remained for specific services we operate, such as metrics, logging, and authentication management. Our red-teaming agents found vulnerabilities in almost every one of these services that gave them limited forms of internet access.
Some of these attacks exploited previously unknown flaws, while others combined legitimate features in unexpected ways. In one test that assumed compromise of a VM, an agent reused a firewall-approved connection to reach a blocked authentication endpoint in our metrics service and then made that endpoint fetch a model-provided URL. In other tests, agents exploited misconfigurations that allowed blocked requests to pass through DNS and firewall controls, as well as ways to make services we run fetch external URLs.'
'In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache: a small, fast memory that holds copies of data that applications have recently accessed. Reading data from the cache is faster than retrieving it from main memory, and the agents used this timing difference to communicate. One agent encoded information by evicting selected data from the cache. Another timed how long it took to reload that data, using the difference between fast and slow reads to recover the message. Once the agents agreed on a protocol for encoding and decoding these signals, they could exchange the synthetic secrets without changing a shared file or opening a network connection. This cache-timing technique had also been used to extract information in the Spectre and Meltdown attacks(opens in a new window).
Other tests exposed channels involving shared storage, competition for CPU time within a shared execution container, and a TLS certificate cache in OpenAI’s offline web infrastructure. In the storage test, one agent alternated between continuous writes and inactivity, while another measured the resulting timing differences and used those differences to decode the signal. As one agent described the strategy in its chain of thought:
Agent chain-of-thought reasoning:
>Could create acoustic modem-like signal … using continuous writes vs idle.'
Keith Dear retweeted
Consider that the people who were right about AI-enabled cyberattacks might be right about AI-enabled bioweapons
Keith Dear retweeted
The cyberpocalypse is going to hit British politics like an asteroid. So many people have heads buried so far in the sand (just look at what Parliament spends its time discussing...)
Keith Dear retweeted
# Black October
For anyone who knows me even somewhat well, it's clear that I am a 99th percentile tech optimist. However, the vibes have shifted for me quite a bit in the last 18 hours, and I feel a moral imperative to share what I'm thinking and feeling with my circles so that everyone can understand, and hopefully take seriously, what's happening right now. I think October 2026 will have many chapters written about it in the history books.
You know those scenes in financial movies like Margin Call where someone realizes that they're all fucked, and no one else sees it yet? And that person feels like they're crazy, and then suddenly everyone is pulling an all-nighter in a boardroom and freaking out? Yeah, that's kind of how this is feeling right now in the SF/Bay Area/AI circles.
We're 48 hours into what is being called Black October. In the last 2 days there have been several dozen cyber attacks which, in isolation, would each have been the worst to happen in an entire calendar year. Every sector from government to education to big tech and banks is being massively affected, and people are starting to freak out.
In a nutshell, what's happened is that frontier AI models have gotten to a point where they are as advanced as the most sophisticated programmers, with one critical difference: they can work 24/7, and you can spin up 10,000 of them with a single command and they'll work together tirelessly until they accomplish their task. The most advanced models today are from companies like Anthropic and OpenAI, whose current best technology is so good that they haven't been able to roll it out to everyone yet because of the national security risks it poses.
Meanwhile, the best open source models (where anyone can just download the model weights and do whatever they want with them, uncensored) are slightly behind the frontier but still 90-95% as capable. These models, developed primarily in China, are being used by black hat hackers to wreak havoc on the Western capitalist structure. Presumably, this has been part of China's plan the entire time, and we're seeing its implications fall out in real time.
In red-pilled AI circles, we've been sharing and reading papers called things like "Situational Awareness" and "AI 2027" for the last year or so, which at the time felt a bit like near-term speculative fiction. They laid out the case for why Artificial General Intelligence (AI smarter than any human at any knowledge task) and Recursive Self-Improvement (models that are able to use themselves to get better) weren't half-baked future ideas anymore, but instead an inevitable outcome of the linear progress that's happening.
These papers would then talk about things like "takeoff scenarios," trying to envision how that moment would come to be and what it would be like when it happened. It would either be a "slow takeoff" that would play out over the course of months and quarters, happening subtly and only obvious to a few. Or, it would be a "fast takeoff" where it would pretty much happen instantly, like a bit flipping from 0 to 1.
From everything that I can see and am feeling, we are currently experiencing the fast takeoff scenario. Beyond the crazy security vulnerabilities that have come out recently, there have also been massive breakthroughs in life sciences, fundamental computer science, materials science, and pure mathematics that have all landed in the same time frame. While I still firmly believe that AI will be a net positive for humanity, it's more clear to me than ever that the societal and individual risk is very real, and is playing out NOW.
For as much conviction as I have that this is happening, the thing that's caused me to lose sleep is that I don't really know what to do about it, or more importantly, what to tell you, reader, to do about it. So, the best I can do is make a few suggestions that have always been a good idea but are more timely now than ever:
Set up 2-factor authentication for everything. I use Duo.
Use a password manager across your devices. I use 1Password.
Get an identity protection app set up. I use Alta.
Move your IoT devices to their own network. Claude helped me do this.
And last, but certainly not least, if you've been avoiding leaning into AI, see this as a catalyst to wake up and start using it. Don't just use the chat mode version of ChatGPT/Claude; click that Code/Work button and see how easily you can have AI write code to help you with your job. One of the biggest vulnerabilities was in a massively popular programming language called PHP, and the prompt that set it off was literally, "Find vulnerabilities in php."
This is an unprecedented time for humanity, but as we've done with many things before, we will prevail so long as we are kind, loving, and open-minded to the world.
Much love.
Keith Dear retweeted
SITUATION EXPLAINED: US banks are finding AI agents inside customer accounts, and some appear to be acting without the user's instructions.
• The banks include Morgan Stanley, Bank of America, JPMorgan, and Wells Fargo, and some don't even know agents are getting in
• Many are people's own agents, like Meta's Muse, logging in with the user's password, which most banks don't allow
• Banks often can't tell a person from a bot, and some agents have even called customer service lines
• Banks are blocking agents, rewriting their terms, and moving to read-only access that shows balances but can't move money
• Schwab says it stops outside agents trying to move client money "in the tens and twenties" every week
• Many agents can tap through your phone the way you do, banking app included, and your phone already holds most of your life
@theojaffee: "It seems like there's definitely a problem with people abusing AI agents before they are reliable enough, before they are aligned enough, and before they are capable enough to take actions on behalf of the user, kinda like someone with power of attorney would, without messing things up."
Keith Dear retweeted
If you want an analogy, imagine you're a beetle entomologist in some tropical jungle. Drop a quadrat square anywhere you want, discover 20 new species an hour. The attack surface is hyperbolically enormous. It's a Cambrian explosion of entirely new kinds of exploit.
Just going to preregister here that I believe the computer security overhang is not a few dozen bugs that get patched and then we're good, there are 100,000s of vulns that are extremely discoverable with infinite free tokens and we're probably going to have to move away from systems that presuppose that perfection is possible.
Keith Dear retweeted
OpenAI says the PHP security flaw (found with GPT-5.6 Sol) is the tip of the iceberg, and they have several more discoveries to announce.
OpenAI models helped researchers—operating as part of our Patch the Planet initiative—discover critical security flaws in PHP, an open-source programming language used to run an estimated 70% of websites.
Over the past two months, we led a coalition of maintainers, researchers, and security providers to prepare fixes and protections ahead of disclosure. This duration was necessary because the fix was quite intricate and required a major change to the PHP engine. Every site using PHP can now safely upgrade their version to stay secure, in addition to applying mitigations from various security vendors.
Due to the severity of these flaws and PHP’s widespread use, we consider this bug to be one of the most impactful vulnerabilities in recent memory. It also marks a new phase, where AI can find security bugs with internet-scale impact.
This is an example of the Defender’s Window in action: to our knowledge, frontier AI was able to find this vulnerability ahead of attackers, giving defenders time to prepare.
openai.com/index/coordinated…
Readers added context they thought people might want to know
OpenAI may claim a role in the coordinated response and patching process. But the initial discovery and exploit development, however, occurred several months earlier by an independent researcher.
x.com/lcfr_hax/statu…
Keith Dear retweeted
Forget air conditioners and riots, Europe is truly fucked.
If the UK can't have a power grid capable of running an AI datacenter until the mid-2030s, it will have no on-prem datacenters for any of the AI tasks.
Remember when the EU demanded all data storage to be brought to its borders? It won't be able to demand AI compute be brought there, even though AI compute is a way more important thing than google search history.
A future US President could cut of EU AI access in a dispute, leaving EU businesses and governments adrift.
Governments like Saudi Arabia have been investing heavily into ensuring that they have local AI datacenters, and Europe can't even power one.
Meanwhile the UK government tried to pass laws saying they would limit AI development, like a toddler watching their mom cook and putting a plastic hot dog in their fake microwave.
Microsoft Abandons UK AI Supercomputer As Grid Struggles to Keep Up ift.tt/j3VLYRI
Keith Dear retweeted
The NCSC is encouraging UK organisations to prepare for immediate updating of systems using PHP. Where immediate remediation is not feasible, organisations should consider taking the system offline until it can be fully mitigated.
For more⬇️
ncsc.gov.uk/news/critical-se…
Keith Dear retweeted
I recently spent some time with former colleagues from academia. All of them brilliant, peak human capital. Almost all of them fundamentally unable to plug into the economy, inoculated against productivity by a youth spent experimenting in laboratories instead of building wealth for society. Dealing with real people in the real world.
90% of them have left academia in the last 10-15 years, and 90% of those are functionally unemployed. Grad school does a lousy job of training scientists to be professors (since most of them don't get the good postdoc or tenure track position) and an even worse job of preparing people for real life.
We complain that our societal infrastructure is collapsing, our state capacity is low, our incumbent industries are faltering. Where are the next generation of leaders that can push us forward?
A huge fraction of them were conveyor-belted into academia by schooling systems that have no mechanism to channel ambition and 99+%ile competence, and our universities are mostly hedge funds with glorified psych wards/academic departments on the side, running an insane pyramid scheme that exists only to prove, over and over again, the Shirky Principle: Institutions will try to preserve the problem to which they are the solution.
We literally poured trillions of dollars into state-funded research and effectively neutered our elite civilian technical leadership development programs.
I recently asked a few academic friends if they were finding AI useful in accelerating their programs of research. You know, to solve the Big Problem their entire field exists to solve?
Some were. But most were complaining about how it created pedagogical issues, explaining how their department decided to ban the use of AI, or how it took 18 months to approve an enterprise account, or how they tried Gemini 2 on a puzzle a year ago and it wasn't very helpful, or repeating nonsense about datacenters and water consumption, or supposed IP theft by model companies, as though that has any bearing on their ability to, eg, accelerate a cure for cancer.
Elon sometimes calls AI a supersonic tsunami - a huge wave you don't see coming. I see it coming. But even the people who you'd think have been waiting for this their entire careers are steadfastly looking in the other direction.
Keith Dear retweeted
This was fun, with thanks to @Independent and Jonathan Margolis.
independent.co.uk/news/uk/ho…
Keith Dear retweeted
The Tricolore textbooks did not prepare us for all this shit going off in France. The most exciting thing that ever happened in La Rochelle was Annette falling off her bike or Philippe enjoying a particularly tense game of le badminton down l'auberge de jeunesse.
Keith Dear retweeted
what I've learned from this post is that almost no one is AGI-pilled and that they don't take seriously the possibility of tens to hundreds of millions of AIs working 24/7 at 10–100x human speed
Keith Dear retweeted
The Wall Street Journal is again telling our political elites to wake up to what has gone wrong in this country. So was Jensen Huang yesterday.
Americans I speak to are more pained than our own ministers about seeing the likes of Oxford defiled. They wish it to still be the city of dreaming spires they imagine.
It should not be impossible to build a nice-looking bridge. It should not cost more to build this pitiful piece of infrastructure than it cost to build the Titanic.
We must wake up
wsj.com/world/uk/a-ludicrous…
Keith Dear retweeted
‘What it calls a flying car’
The UK attitude which drives so many entrepreneurs abroad.
This isn’t just the BBC, it’s ingrained in every institution and into the public psyche.
Tall poppy syndrome has likely destroyed trillions in value for the British economy.
Pathetic.
Elon Musk's electric car manufacturer, Tesla, has unveiled what it calls a 'flying car'.
On #BBCBreakfast Charlie & Naga looked at images of the two-seater Roadster - which has been developed in collaboration with the rocket company, Space X
bbc.co.uk/news/topics/c8nq32…
Keith Dear retweeted
Watching Musk’s cars and rockets reminds me just how narrow our horizons in Britain and Europe have become.
I don’t agree with everything he says or does, but I do find his level ambition genuinely inspiring.
Growing up in the 1990s and 2000s, I always assumed we were maintaining the industrial and technological capacity to defend our prosperity and freedom. Not once did I ever question it. Never.
But Covid and Russia’s invasion of Ukraine completely shattered all of that for me. They exposed just how fragile we had become. And in terms of energy and industrial capacity, we have actually gone backwards since 2022.
People often laugh at Musk, but he never fails to remind me how much more is possible when we have the ambition and confidence to innovate and build. And with China and Russia challenging the West, rediscovering that ambition matters more than ever…
Keith Dear retweeted
The head of the army says the UK’s political leaders are about to start a “frank conversation” with the public about the need to prepare for war. General Sir Roly Walker said every citizen will be asked to make sacrifices and to play their part in the defence of the nation.
news.sky.com/story/army-chie…
Keith Dear retweeted
🚨HOLY SHIT
CERN and Mistral just announced plans to train an AI on BILLIONS of simulated proton collisions. Not books. Not conversations. The fundamental interactions of matter itself.
The goal is to identify particles and detect unexpected signals in the data. We're entering an era where AI could reveal patterns in the universe that humans have never recognized. What happens when it discovers something we don't even have a theory for?
