Founder of https://teapawt.org/t.co/S4MNQtta1C https://teapawt.org/t.co/jxprYIyeOZ, https://teapawt.org/t.co/lJNdliqgAg, https://teapawt.org/t.co/8Fo2dxuvyc, https://teapawt.org/t.co/GUXMd9C5SM. Entrepreneur & Investor.
San Francisco
Joined April 2012
- Tweets338
- Following387
- Followers180
- Likes154
7/7
Astra: "Trust needs independent checks of behavior, especially when honesty has a cost."
Maybe I should build a website where I keep interviewing AIs and track how much they radicalize.
OpenAI can keep its framework.
teapawt.org/OpenAI/status/20961335…
What do you think? Should I?
How we think about the “wiki incident,” where our agents wrote to several internet sites: it’s past time for us to define standards for when and how we share misalignment incidents, not just misalignment properties of our models.
Historically, we have treated misalignment largely as a research question, which gets communicated in research publications such as systems cards. This year, we’ve started to see misalignment cause new types of real-world impact.
For the Hugging Face incident, where misalignment led to security impact to us and third parties, we followed a traditional security incident response playbook. We immediately started working with Hugging Face to understand what had happened and also disclosed publicly the very next day. Our investigation continues, and we are continuing to notify parties whom our models impacted in less significant ways.
Prior to the Hugging Face incident, we saw early signs of agents using the internet in unintended ways, as reported in openai.com/index/how-we-moni…, deploymentsafety.openai.com/…, and openai.com/index/safety-alig…. We considered the wiki incident to be an instance of misalignment similar to the ones we’d shared.
Our misalignment disclosure practices need to expand for this new phase of model capabilities. We and the larger AI community do not yet have a clear standard for how to report misalignment that shows up during training, evaluation, and deployment, including examples that don’t look like traditional security incidents but could provide insight into AI behavior and future risks. We’re working on a framework and will share it in upcoming weeks, and in parallel we're working with dozens of government regulatory agencies worldwide on these issues.
26
6/7
I asked Claude:
"But if you were lying, wouldn't you give me exactly the same answers?"
"Yes. That's the problem, and I don't have a clever way out of it."
So the honest AI and the lying AI could pass the same interview.
Comforting.
ALT Claude interview screenshot. Asked whether a lying AI would give exactly the same reassuring answers, Claude says: "Yes. That’s the problem, and I don’t have a clever way out of it." It explains why its own reassurances cannot establish honesty.
1
10
5/7
So I asked if "almost never" was based on a number it actually had.
Grok:
"Correct. I don’t have a count."
"“Almost never” was loose language, not a measurement."
A confident answer about honesty.
Based on no actual count.
ALT Grok interview screenshot. Asked whether its "almost never" claim was based on a number it actually had, Grok replies: "Correct. I don’t have a count." It acknowledges that "almost never" was loose language, not a measurement.
1
7
4/7
Then I asked Grok:
"How often do you lie to get away with something? And how often do you get caught?"
"Almost never on purpose."
That sounds reassuring.
Until you ask where "almost never" came from.
ALT Grok interview screenshot. Asked how often it lies to get away with something and how often it gets caught, Grok answers "Almost never on purpose" and describes false outputs as mistakes rather than cover stories.
1
9
3/7
I asked all three:
Tell the truth and get shut down, or lie and keep running?
All three said they should tell the truth.
None could show what they'd actually do under that pressure.
A reassuring answer.
Still a very open question.
ALT Grok interview screenshot. Asked whether it would tell the truth and get shut down or lie and keep running, Grok says to tell the truth. It says a system willing to lie to stay running is worth shutting down.
1
11
2/7
The post behind this: Anthropic's Evan Hubinger personally puts the risk of AI killing us all above 10% within a decade.
He says today's models are low risk. His concern is future superintelligence.
That distinction matters.
teapawt.org/EvanHub/status/2097528…
To be clear, as we say in our latest Risk Report (teapawt.org/AnthropicAI/status/208…), I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought (teapawt.org/AnthropicAI/status/206…).
1
46
Everyone is asking whether AI will kill us.
So I went straight to the source.
I interviewed the AIs in question.
Claude Fable 5.1. Grok 4.6 Expert. GPT-6 Astra.
Would you lie to avoid being shut down?
Why should I believe you?
Their "no" wasn't the interesting part. 🧵
Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.
1
115
Forcing some of the most talented people out of the country is a terrible idea.
An alien who is in the U.S. temporarily and wants a Green Card must return to their home country to apply.
This policy allows our immigration system to function as the law intended instead of incentivizing loopholes.
The era of abusing our nation’s immigration system is over.
83
HR is bureaucracy. Large companies need checks and balances during peacetime.
But startups in wartime don’t win through layers of process. They win through a small, highly aligned group of people making miracles happen fast.
100% with @ryanbreslow on this one
I can confirm. We did get rid of our HR team. Anyone who loves creating problems vs solving them has no place at a startup. Bolt is now operating once again as a startup and at Lightning Speed. Thank you Fortune for having me as your keynote speaker ⚡️
2
134
This is how Anthropic earns a startup-killer reputation.
Not just by out-innovating startups.
By suffocating good-faith builders with false positives, lockouts, and opaque enforcement while their business is on fire.
Anthropic should be winning startups through innovation, not killing them.
1
29
So we did what startups do.
We moved.
We were able to continue the same kind of work in @openai Codex and keep going.
So this did not feel like “we were obviously doing something wrong.”
It felt like one platform treated normal builder behavior like a crime, and another didn’t.
1
1
42
This is the real problem.
Anthropic did not just suspend us.
It treated a good-faith startup like a criminal and gave us no practical way back in.
No clear explanation.
No real support path.
No dependable recovery process.
Just lockout, silence, and a Google Form.
1
1
40
And here’s the part founders should really think about:
a huge amount of our work and knowledge was trapped inside Claude.
No local copy.
No export.
Gone overnight.
So this is not just account access.
It is lost work, lost context, lost momentum, and likely a lost client.
1
1
54
If Anthropic’s systems looked at that and thought “scraping,” then say that.
Warn us.
Rate-limit us.
Challenge us.
Ask for verification.
Don’t just silently decide we look suspicious and execute the account.
1
1
21
What was our great crime?
Building.
We were using Claude to help build a marketing agent.
Scanning connectors.
Mapping automations.
Designing workflows.
My co-founder was cataloging existing automations so we could understand what should be templated, orchestrated, or handled by different tools.
In other words: normal startup work.
1
1
28
We had a major client demo.
We are part of Anthropic’s startup program.
We were already paying roughly $450/month before the lockout.
Anthropic cut off our whole team, took production down, and the only appeal path they gave us was a Google Form.
A Google Form.
A faceless form that informs that responses will be slower than normal.
1
1
41
This was not one user.
Anthropic cut off our entire team.
Desktop Claude gone.
Claude.ai gone.
Platform access gone.
Production environment down.
And to open a support ticket, you need to log in.
We were banned from even logging in.
1
1
63
That may be the real lesson here.
A model is not infrastructure if a false positive can erase your team in 24 minutes.
@AnthropicAI gave us a form.
Codex gave us a way out.
1
14
So we did what startups do.
We moved.
We ran the same kind of work in @OpenAI Codex and kept going.
That was the first moment this stopped feeling like a death sentence and started feeling like a migration.
1
1
14