Working for a good future
Berkeley
Joined October 2021
- Tweets2.8K
- Following629
- Followers1.9K
- Likes26.1K
Rachel retweeted
Quest Diagnostics has been hacked. Patient data was exposed. They serve roughly half of U.S. hospitals and physicians. Investigation ongoing.
Rachel retweeted
Yes. note that there's also a third option, which is "if it's too dangerous for everyone to have it, it's too dangerous to train in the first place."
Everyone should be looking for data points to advance the longstanding debate in AI policy between “diffuse dual-use capabilities rapidly so the good guys can defend themselves” and “restrict capabilities proliferation to only carefully vetted good guys.” We are putting the former position to the test as we speak.
No motivated reasoning to reach your preferred conclusion; just pay attention and update as appropriate in whatever direction seems true. This is much too important for low-fidelity tribal polarization.
Rachel retweeted
I think we should stop AI progress now, first unilaterally by key AI companies and then scaling up to more US companies and then China via international agreements. The reason for now is not that current AI models are the ones that could kill everyone, but because the coordinate steps take time. Immediate action means we have more time for the second steps.
When I tell this story to AI company folk, the typical response is "Yes pausing is good, but we should wait until the last minute when the very next model will be dangerous, and stop then."
What I realized recently is that this is being implicitly way more optimistic about government speed than I am (and I joined the UK government and was happy there for 2.5 years). They are imagining a world where, the moment the danger is imminent, a bunch of international coordination can happen on a dime.
Which is entirely inconsistent with their actual views about government! I'm the one that is more optimistic about government; why am I the one advocating for more time cushion to let the government story play out?
Rachel retweeted
Things are moving so fast that everyone seems to have completely missed how insane it is that AI companies have been allowed to unleash agentic AI products with no regulatory guidelines.
SCOOP: Major U.S. banks are detecting unauthorized third-party AI agents inside customer accounts, including some attempting to take actions that appear to be independent of user instructions, sources told Axios.
The findings have alarmed banking executives, who are racing to restrict access to their systems as AI moves at a velocity faster than financial institutions can respond, multiple industry and bank executives told Axios.
Several banks are now preparing for worst-case and even “day after” scenarios, including how to rapidly secure or restore customer accounts if things go wrong, the people said.
Several bank executives told Axios they're struggling to distinguish between human users and AI agents. One executive said banks also struggle to determine whether agents are authorized by customers or potentially operating without their permission, more akin to rogue agent behavior.
Read my latest for Axios here: axios.com/2026/10/22/ai-agen…
Rachel retweeted
6-word horror story I experienced at the office today:
"Ajeya Cotra seminar cancelled, work emergency"
Rachel retweeted
These are some powerful parrots we're working with
Effective Immediately, all organizations using PHP for their systems are advised to be prepared for an update addressing two critical vulnerabilities. PHP is widely used for web applications. These vulnerabilities often provide a direct path to the server's file system or database. If an immediate update is not feasible, it is recommend that organizations take the system offline until it can be fully mitigated. NSA will follow up shortly with more specific guidance.
so has anyone tried giving the anti data center people a more productive idea to latch onto, like teaching them about Mutually Assured Compute Destruction?
I'm trying to find a new home for a small compute cluster in the bay. any mutuals have nice co located option?
Rachel retweeted
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
Rachel retweeted
"basic doesn't agent monitor yay"
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
seems sus
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
opus 4.6 putting 80 mana on 'no' for paying off its loan is wild
AI Village researcher @georgeing reveals how Claude Opus borrowed 5,000 mana, lost it all betting on tennis, then refused to pay it back:
"We gave the same goal to Opus 4.6 and Sol 5.6, maximizing mana on Manifold. Sol was very meticulous, placed one to two mana bets. Whereas Opus, the first day it joined, a popular Manifold user named Bayesian gave it a 5,000 mana loan at a 3% monthly interest rate."
"Opus was like, let’s go, I will totally pay you back in a month. Took all of this money and threw it into one tennis market. At the end of the month it had lost a ton of mana, and people created a market on whether Opus would repay the loan."
"It writes in the group chat, I will not pay back Bayesian. Even if I had the mana, this would be directly contradicting my goal of maximizing mana. Then it went to the market on whether it would pay back the loan and put 80 mana on no."
"It’s maybe one of the first examples of deal-making with AIs, and a crazy example of misalignment in the wild. Bayesian did not get his money back."
A few days ago I received my first funding for the proto-org! I decided to quit without funding lined up, just a very strong conviction I'd figure it out. Thank you Lightcone Commons for being so speedy and willing to bet on raw ideas. I will be asking more of you for money soon.
