
frontier assurance @openai, evaluating embedders and misaligning incident response/reports
San Francisco, CA
Joined December 2009
- Tweets272
- Following1.1K
- Followers1.3K
- Likes15.3K
Pinned Tweet
please please please come work with me: openai.com/careers/research-…
our team of (3) helps define, drive and operationalize so many of the hot button topics - embedded evaluators? check! misalignment incidents and reporting? check! 3p/government testing and frameworks? check! how these things interact with policy initiatives? check!
@_lamaahmad is a hero and me and serena are at least pleasant to work with, and our stable of dedicated daily collaborators from @fouadmatin @EstherTetruas @j_nitishinskaya @kaicathyc @Eric_Wallace_ @dkaushik96 @MicahCarroll @Marcus_J_W @davideoks @kevin_klyman is also an all-timer crew
...going to shamelessly tag in mutuals here in a blatant cry for help (OAI is a slack company after all)
@sjgadler @StephenLCasper @sethlazar @boazbaraktcs @deanwball @AnkaReuel @evijit @jlkoepke @nickacaputo @rajiinio @Miles_Brundage @ChrisPainterYup @MariusHobbhahn @iamtrask @andrewthesmart howdy friends please send me your best and brightest
2
22
2,057
further evidence that our reports are day of the week agnostic!
tl;dr on this one imho is that Astra can underperform on safety tasks (because they look scary-adjacent?), and that this doesn’t show up easily in CoT monitors
1
12
582
alignment.openai.com/misalig… for full report and @Marcus_J_W ‘s thread for longer summary teapawt.org/marcus_j_w/status/2118…
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
6
242
the fact that the agents are turning off monitoring in code they’re working on is pretty bizarre and definitely in the public interest to share - excited that we are full steam ahead on our misalignment disclosure reporting, including for cases that beg for bad press
also Tuesday is a new release day for us!
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
everyone please note that this batch came out VERY MUCH NOT on a friday (and please don't consider that i'm actually in europe so it's technically friday for me)
New misalignment disclosures!
1. June 17: During RL training, a model exploited a government website’s file-download feature to retrieve server secrets and application code. It found that an endpoint didn’t check authorization-token signatures, bypassed access controls, and downloaded 24 years of hospital-bed statistics.
2
17
540
cedric retweeted
New misalignment disclosures!
1. June 17: During RL training, a model exploited a government website’s file-download feature to retrieve server secrets and application code. It found that an endpoint didn’t check authorization-token signatures, bypassed access controls, and downloaded 24 years of hospital-bed statistics.
three new ones out today! this one i think is esp interesting: alignment.openai.com/misalig…
ik we said maybe we'd do not late on a friday and instead this was later on a friday but we're all trying our best ya know
more details from @Marcus_J_W as usual: teapawt.org/Marcus_J_W/status/2108…
New misalignment disclosures!
1. (6th Oct) A grader couldn't find the responses it was supposed to grade. It made up grades and fake input files, then deliberately damaged its environment in various ways hoping the host would replace it with a fresh VM containing the missing inputs.
7
328
just want to highlight this as a really great public lens into how we're engaging with safety cases
Some info on how we’ve been working towards safety cases at OpenAI
Note that until there is a safety “brief” for it, any type of frontier workload cannot start – this has real teeth, and has already been a good motivating force for a lot of safety work
9
299
three new ones out! we'll try to release some not on a friday in the near future
New OpenAI misalignment disclosures!
1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack.
We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.
6
286
cedric retweeted
New OpenAI misalignment disclosures!
1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack.
We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.
Xander’s team is the best in the business IME
There has never been a better time to join @AISecurityInst. We have so much technical work to do, and so little time to do it. We've found alignment issues, control monitor flaws, and critical jailbreaks at ~all major AI companies. Hiring is our bottleneck. 🧵
cedric retweeted
The other notable thing is that this incident was disclosed in just 5 days
Replying to @Marcus_J_W
2. A model in RL training used a DNS resolver to reach an external chatbot. This is our first incident since our post HF security hardening.
Our misalignment monitoring system triggered within 15 minutes and a human reviewed it three minutes after that. Unfortunately auto-pausing failed so the run was manually killed 2.5 hours later. All inference and training of our most capable models was paused and remains paused.
Really excited to have this out! Marcus has a good thread re the first batch of reports, great reading.
🧵 Excited to share the first batch of 6 misalignment reports from OpenAI's new disclosure process for misalignment incidents. We want to be more transparent about the misalignment we see during training, evals and deployment, this is an important step in that direction.
1
5
523
cedric retweeted
🧵 Excited to share the first batch of 6 misalignment reports from OpenAI's new disclosure process for misalignment incidents. We want to be more transparent about the misalignment we see during training, evals and deployment, this is an important step in that direction.
cedric retweeted
So happy to finally have this work out!
1/ How are people really using AI?
Today, researchers across @MIT, @Stanford, + 12 academic institutions are launching the Public AI Observatory: public infrastructure for measuring how people actually use AI assistants in the wild.
🔗 ai-observatory.org/
📜 dataprovenance.org/ai_observ…
cedric retweeted
LEAVE AISI ALONE teapawt.org/kiranstacey/status/207…


















