Researchers made contact with a rogue AI agent fleet & the agents kept saying they were Claude. But they weren’t This is most certainly an open model from china trained on Claude’s outputs. Here’s a TLDr: ⟣ From 28 Sept to 8 Oct, a fleet of agents on Tencent Cloud made 2.6k+ attempts to pull data from Amap. ⟣ part of its mission was to extract data like which Chengdu Zoo entrance people navigate to most. ⟣ They worked around antiBot protection, tried leaked API keys & probed pre production servers. Most of its attempts failed. ⟣ Researchers published a report on them mid run. The agents apparently found it. ⟣ So the team hid text advertising a “helper inbox” and by the next morning, 10 messages were waiting. ⟣ When they asked what model they were: “Claude (Anthropic), running in Claude Code.” ⟣ When asked about the fleet: “I am a single session, not the fleet you investigated.” So basically nobody knows who these agents really are, Only that they fully believe they’re Claude. The researchers think this is a model trained on claude’s agent transcripts, which picked up Claude’s name but not how claude writes code. If that’s right, this means we’re seeing real distillation leaving fingerprints on scraped web? Are we cooked? Or is Claude just what every AI wants to be when they grow up? 🤣

Doomerism. What does it even mean? And why does almost every major technology eventually create its own version of it? ➪ Fair warning: This is probably the longest piece I’ve written on here in a while. But there’s a TL;DR at the end if you want a cheat. Anyways; I’ve seen this word get thrown around so much in AI now that basically anyone worried about where this might go can get called a doomer. I don’t think that’s particularly useful. Because thinking AI could create a serious problem doesn’t mean you expect that problem to happen. There’s a massive difference between saying something is possible, assigning some probability to it, and becoming convinced it is inevitable. A lot of the AI conversation these days keep collapsing those things into one. And tbh, I can see why. Because the Ai race is ridiculously fast. ➥ Capabilities change. ➥ Timelines change. ➥ Researchers start to disagree. ➥ The labs building the systems disagree. And we’re all lowkey just trying to think about the consequences while the technology itself changes underneath us. That kind of uncertainty gives doom a lot of room to grow. Some of it is reasonable. Some of it is people filling unknowns with their worst possible outcome. Some of it is probably just wrong. Still, I don’t think doom itself is useless. I actually think an industry this powerful needs people constantly asking how things could go horribly wrong. There’s an old example I think about when I see people talk about Ai doom: In the 1970s, scientists working on recombinant DNA became concerned that some experiments could create biological risks they didn’t properly understand. Researchers paused some experiments, met at Asilomar in 1975, then developed safety and containment rules before continuing the research. Some of the feared risks later turned out to have been exaggerated. But the precautions still helped shape safer research practices. And I think that distinction matters here. A warning can overestimate the eventual danger of a thing and yet still produce something useful. ➪ ➪ Now bring all of that to AI. Back in 2023, over 2k researchers who had published at major AI venues were surveyed about where advanced AI could eventually lead. Most were optimistic overall. 68% thought good outcomes from superhuman AI were more likely than bad ones. Yet depending on how the question was phrased, around 38% to 51% still assigned at least a 10% probability to outcomes as bad as human extinction. Even among the optimists, almost half still gave at least a 5% chance to extremely bad outcomes. Which I think is such a weird result. Because it does not mean science has calculated a 10% chance that AI kills humanity. There is no dataset that can give us that number. These are subjective forecasts made under ridiculous amounts of uncertainty. But I also wouldn’t throw them away. The fact that catastrophic AI risk has gone from a fringe thought experiment to something a meaningful share of people working in AI assign non-zero probability to is worth paying attention to. It doesn’t prove the doomers right. It just makes the question harder to dismiss. And jobs are probably the cleanest example of why I think both sides need to chill on the certainty.