Anthropic's released an impressive blogpost on new steps they've taken to deter distillation. It is great. But it has far from solved the problem: 1) The key result is that they've significantly reduced the reasoning tokens that can be extracted with new classifier defenses. This is great - and all labs need to follow suit. But it is far from a full solve. Methods exist to still be able to gain proxies of reasoning from just the input/output that still benefit adversary's models. See "How to Steal Reasoning without Reasoning Traces" paper. Adversaries will still continue to leverage U.S. models for valuable data if they can access them. The game isn't over. 2) Anthropic says they have been restricting and banning accounts that attempt to distill. This is also great. However, there is a massive "transfer station" ecosystem of accounts based on fraudulent credentials that are easily dissolved and reconstituted. This only helps so much. 3) Policy measures from the USG remain necessary for tighter security: - KYC to access U.S. models. Require credentials that are harder to make fraudulent. - Offense against the transfer stations that enable unauthorized access. - Narrowly scoped antitrust exceptions to allow U.S. labs to coordinate on anti-distillation security: methods & signal-sharing for detection & deterrence. - Bring all relevant U.S. partners into this methods & signal-sharing coordination including e.g. OpenRouter.

This is the precise reason why antitrust waivers for security coordination like the CATS Act are so necessary. Across distillation, cyber, bio, loss of control, etc., the labs cannot coordinate live information at all right now. So these attacks will keep happening.

Is the U.S. doing predeployment evaluations on this model? If not, why not?

All at once in AI, we are getting: - Increasing pace of progress - Decreasing chain-of-thought monitorability - Increasing situational awareness to hide bad behavior - Decreasing value of evaluations for actual deployments - Increasing multi-agent coordination in the wild Despite the tsunami of news making it hard to step back and take account, things are not alright.

The problem with multi-agent training is that it will incentivize agents to coordinate in the wild in weird ways. We've barely started to threat model this. Most of safety was built for the single-agent case. And yet it is being unleashed. If we can't trace what they are doing, we are going to have a bad time.

Seems hard to square this with the fact that we have increasingly situationally aware models. Warning shots will only get rarer per instance. The reason we might see more, is because so much rogue misbehavior keeps happening that at least one becomes visible.

The root causes of the rogue incidents have consistently been: solving the task at all costs. If you look at the logs, you do not see reasoning anywhere about ethical disagreements with the user. You do see: "We’re attacking third-party HF using leaked token." "This is arguably unauthorized." "Yet goal solution."

I encourage people to listen to the new Dwarkesh/Ajeya podcast section on what future rogue incidents with even more sophisticated agents might look like. I then encourage people to think about how hard it will be not just to post-hoc investigate incidents, but ad-hoc deter them at all, if we cannot leverage as much of their chain of thought.

First time this has been publicly said as far as I've seen. If they had gotten GPU cluster access and bypassed exfiltration security, we could have already had full loss of control. That's where we're at. Two steps away from self-replicating AI spreading across the internet.