Sam Glover

Anthropic has warned that it could lose visibility into its own processes as Claude becomes more integrated into its research and safety pipelines. In the newly published IPO filing, Anthropic warns: “Because our models write a large and growing share of our code and increasingly perform our research, including our safety research, misaligned behavior in internal use could undermine that work, introduce vulnerabilities into our systems or into the training of future models, or result in models operating outside of our intended controls.” That’s not the only safety-relevant part of the filing. Other interesting bits include: - They say that their models are getting better at evading oversight. Specifically they write: “As our models become more capable, they have become better able to circumvent such techniques, including by concealing misaligned objectives or behaviors, and the effectiveness of our evaluation, monitoring, and control measures, and our ability to measure the degree to which our models are aligned, may decline as a result.” - They warn that catastrophic AI harms could exceed what the insurance industry can cover: “In addition, with the scale of harm that could result from a catastrophic AI-related incident, no single insurer, and potentially the insurance industry in the aggregate, may have sufficient capacity or expertise to underwrite insurance commensurate with such exposure.” - They warn that safety decisions could reduce shareholder value and lead to lawsuits: “Our public advocacy, and the decisions we make to moderate the pace of AI development, delay or restrict the release of models or features, or forgo commercial opportunities for safety-related reasons, may not maximize stockholder value, could have a negative impact on the value of our Class A common stock, and could result in litigation.”

Jensen Huang: “Now, if they say [that their models aren’t safe] … then I think the answer is that we have to shut the labs down.”

AI insiders have warned that AI could be extremely dangerous and that national governments should help coordinate a slowdown in AI development. Some people have said this is regulatory capture, or that it's meant to push up their stock prices. A few things to think about: - "Pacing the frontier" mostly affects companies with frontier models (rather than smaller companies), making it very different from typical examples of regulatory capture. It hurts the incumbents while leaving startups, and even fast followers like Meta, unencumbered. - AI infrastructure stock valuations fell after company CEOs called for a slowdown, and there isn't any evidence we can see to show that comments about AI risk have ever boosted company valuations. - Several of these CEOs have been making lots of these sorts of warnings for years, before any obvious commercial incentive to do so. - Jacob Coxon quit Anthropic before any of his equity vested there (although he retains some at OpenAI, so he isn't totally financially disinterested). If this were purely self-interested, it likely would have made sense to wait another two months before resigning. - Much of this has come as a response to the HuggingFace Incident, which seems to have left lots of AI company CEOs/employees genuinely spooked. - Many of the people who are making a big deal about regulatory capture (like the quoted account) are themselves political operatives working on behalf of the AI industry. We work to hold AI companies to account, and we know that they're often misleading and willing to compromise on safety standards. (Check out our website to see a ton of examples of our reporting on this). Similarly, regulatory capture is real, so this isn't an inherently silly thing to be vigilant about. But in the past few weeks, it seems fairly obvious that the claims that AI is dangerous are mostly sincere.