Joe (@joemkwon) header
Joe (@joemkwon) profile picture

Trying to contribute to better (AI-entangled) futures! https://teapawt.org/t.co/i0NsmZ4J4C Previously @forethought_org, @GovAIOrg, @LG_AI_Research, @MITCoCoSci

Washington, DC
Joined March 2019
Photo by Joe (@joemkwon)Photo by Joe (@joemkwon)Photo by Joe (@joemkwon)Photo by Joe (@joemkwon)Photo by Joe (@joemkwon)Photo by Joe (@joemkwon)Photo by Joe (@joemkwon)Photo by Joe (@joemkwon)Photo by Joe (@joemkwon)Photo by Joe (@joemkwon)
Man, reading through the details more, I'm super disappointed with where OpenAI landed on this. If OpenAI actually believes this is "employee-like" as they promised, I would love to see them justify this. It really seems like they reneged. To be clear, I'm still more frustrated with Google/Meta/xAI, which as far as I can tell have still said ~nothing after signing the White House Accord way back in September (!) which pledged to partner with external groups. But I had really hoped for stronger from OpenAI. 1) The access does not include Slack channels, which is where a huge amount of the context, debating, concern-raising, etc., happens, as I know from my own time working there. Obviously no safety employee at OpenAI would be content with lacking Slack access; it is maybe the #1 place where work happens. This is terrible for trust-building with the public; OpenAI basically preserved secret channels for staff to communicate about safety concerns without the assessors being any the wiser. I know many of the people involved at OpenAI and don't think they have any intent of running a conspiracy like this, but you have to consider how this will look. And more importantly, it is in fact going to hinder the quality of oversight that third-parties can provide. We know from public reporting that OpenAI employees had discovered multiple rounds of warning signs before the Hugging Face attacks. Would this level of third-party access have caught those important details, and the discussions that surely happened over Slack about why they were or weren't concerning? I certainly don't think so. 2) OpenAI seemingly controls what staff the assessors can talk to, about what topics. This is not at all how being a safety employee at OpenAI works. Imagine not being able to talk to another safety teammate about safety concerns unless you get permission from the higher-ups? Assessors should be able to talk by-default with anyone who is safety staff, not need them to be named one-by-one. And if a safety staff member is concerned enough to reach out to the assessors despite not being authorized, that seems like a giant red flag. It seems wild to me that assessors then seemingly need to tattle on staff who raise those concerns? But maybe I'm misreading this teapawt.org/sjgadler/status/212240… 3) It seems like OpenAI's assessments are much weaker than what Anthropic has agreed to. One example is on the ability to share information with assessors. For instance, Anthropic says “Anthropic employees are encouraged to share information with [METR’s assessors] just as they would with a fellow employee working on alignment risk.” There are some carveouts for not sharing obviously sensitive IP, but it doesn't seem like they restrict the particular employee list. OpenAI, in contrast, seems to require a strict pre-authorized list, as I described above. And OpenAI's assessments seem super narrowly scoped, relative to what I've understood of Anthropic's. (If I'm wrong about this and Anthropic is restricting the employee list too, that's important to know and I hope it'll be clarified.) Overall, I've so far talked to exactly 0 people who consider OpenAI's arrangement to be employee-like. I'd love to see OpenAI explain why they believe this. I'm glad that so many inside the company have worked so hard to even get to this point, but IMO it is clearly inadequate for the stakes ahead.
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. openai.com/index/our-first-p…
9
15
2
115
9,148
Worth noting: OpenAI is far-and-away the most consistent and thorough on reporting incidents, IMO. I'm confused what other labs are waiting for. Your researchers believe there is serious danger on the horizon; the way to convince people is to share evidence. If you aren't seeing incidents like OpenAI is, that would be very helpful to know as well. You could even do something less rigorous than OpenAI's process and that would still be helpful. OpenAI's reporting doesn't meet the full Guidelight Transparency standard, to be clear - more to say on this shortly - but other labs seem more behind, and I'm grateful OpenAI is doing this. (As far as I can tell, Anthropic last published incidents in October, which happened September at latest; have there not been any incidents since then? It's possible I'm missing something though, since I'm not aware of a central hub like OpenAI has.)
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
6
33
5
359
28,058
IMO reporting misalignment incidents is crucial for helping us figure out risk levels and iterate towards better practices - my talented colleagues at Guidelight have put together a taxonomy to help guide what to look for and how to do reporting!
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
1
2
15
772
Misalignment is a fundamental driver of loss-of-control risk. We may not be able to rely on visible signals of misalignment for long, but today they're some of the best evidence we have.
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
1
2
5
516
Excited about the work that former lab employees are doing at Guidelight
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
1
3
27
1,487
Guidelight is one of my very favorite AI safety orgs working today. More people should be familiar with their work trying to outline standards for how AI companies should be approaching these risks. All of them need to do far far far better. Check out their latest
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
2
6
95
4,465
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
4
22
11
127
38,012
Hot take: AI lying to you is obviously bad. Other misalignment is more subtle though, which is why we’ve published categories worth monitoring for. We’d love any feedback!
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
3
4
1
39
4,855
We badly need a better science of misalignment. Often it’s unclear what to even look for. We’ve published 10 categories of AI misbehavior for developers to detect and report! This would help us study the root causes of these behaviors and how to prevent them.
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
2
5
20
859
OpenAI's Control Room seems like a great step forward, but I'm not convinced it's actually solved the problem: I worry OpenAI is still vulnerable to AI interfering with its own monitoring. To be clear, I suspect other AI companies are even worse on this. In Guidelight's analysis, we've found no evidence that anyone has this problem handled. And IMO OpenAI is doing a big service by describing Control Room so thoroughly. But after reading it all, I have three main concerns. 1. OpenAI's measurement of Control Room's reaction time is not the most important measurement. Specifically, OpenAI describes the "maximum time" as being 35 seconds between when Control Room is told to stop certain work and when the work stops. This seems impressive and I'm glad they've gotten it so low. But fundamentally this is only reporting one part of what matters, and I fear is too easy to misunderstand. 35 seconds is the time-to-intervene *once a decision has been made* to intervene. But what would be more informative is the time-to-intervene from `the bad behavior itself.` (That is, counting the time before the possible bad behavior _even reaches a monitor,_ & then the time to scan it and actually make a decision.) An even better implementation would be using a `blocking monitor`, so that bad behavior is prevented from taking effect in the first place (albeit still not perfectly), rather than hoping you can intervene quickly enough afterward. OpenAI commonly frames its monitor systems this way (speed-to-act-once-flagged), which feel to me like they're emphasizing the wrong thing. For instance, OpenAI has described sending "highest-priority security alerts" for serious misbehavior, which reach its team members "typically within 15 minutes of the initial alert" when paging is enabled - but this again is not telling us the time since the behavior itself. How long does it take between misbehavior and the initial alert being issued? At the end of the day, OpenAI still seems to be doing 'after-the-fact monitoring,' just a pretty quick version of it. But this has major vulnerabilities. What if AI takes actions that interfere with the monitoring or with the priority alerts to their team? This can cut OpenAI's monitoring off before it gets to act, even if the monitor otherwise would have been very fast to intervene. 2. I'm worried OpenAI's Control Room is still brittle to an AI interfering with it. OpenAI's response about why 'after-the-fact monitoring' is ok would likely be that their inference systems "fail closed" if Control Room goes down. This is a good start; Guidelight recommends something similar in our Control standard. But OpenAI's implementation still seems likely brittle, based on the details they've shared. For instance, if AI can make changes to Control Room or to the engine inference code, it might be able to make changes that *turn off the failing closed.* (To defend against this, I'd want making changes to be gated by a blocking monitor.) Or, the AI might not need to turn off Control Room, but merely make it return dummy responses, so that everything looks ok to the inference engine. I could imagine a version of Control Room that does some hard-to-spoof process, and so is harder to inconspicuously tamper with, but OpenAI doesn't describe this. Right now, I don't have confidence that Control Room has defenses against tampering that leaves it apparently intact but not actually doing its role. OpenAI does say though that Control Room is robust to disabling that is due to "tampering by a bad actor." It's possible this is true, but I don't see the evidence of it. 3. I wonder whether jailbreaking could undermine the efficacy of Control Room. One confusing thing to me: OpenAI says that when Control Room has marked certain work as stopped, OpenAI's engines "must refuse the inference request" and so the engine "returns a refusal." This reads to me oddly, as if the engine has some latitude in how to respond. Why doesn't this just automatically return an error code - rather than relying on the engine to refuse? Maybe that is indeed what happens, and it was just worded ambiguously? I worry that OpenAI's wording implies that jailbreaking could be a viable way of getting engines to continue producing useful output, even once Control Room has meant to stop some work. Though admittedly I'm not sure of this; the phrasing seems unclear to me. ~~ Anyway, I appreciate OpenAI sharing so much about its systems, and I wish that other frontier labs would do the same. Ultimately though, I am not confident that any AI company is where it needs to be on their monitor system quality, and I fear we are playing with fire here.
We built Control Room, a central system to track model workloads across our research infrastructure and stop them quickly when safety or security concerns arise. It connects every research inference workload, including agentic sampling for model training and evaluation, data processing, and more. This works together with our misalignment monitoring systems, and halts model output within 35 seconds of a decision to intervene to stop or pause a job. Control Room is just one part of our holistic approach for safe frontier model training and deployment. openai.com/index/control-roo…
6
6
1
57
6,365
We've posted a new short paper online: What if automating AI R&D triggers an intelligence explosion? The paper assesses the evidence for a software-driven intelligence explosion and recommends high-level policies to prepare. It's coauthored with leading academics, senior scientists of frontier AI companies, and independent experts from civil society. casp.ac/reports/intelligence…
Could automating AI R&D radically accelerate AI progress in an “intelligence explosion”? Preliminary evidence suggests that it could. In a new paper with authors across academia, civil society, and frontier AI companies (including @dawnsongtweets @merettm @jackclarkSF @Yoshua_Bengio @geoffreyhinton ), we assess the evidence & offer policy recommendations 🧵
2
4
13
1,461
Could automating AI R&D radically accelerate AI progress in an “intelligence explosion”? Preliminary evidence suggests that it could. In a new paper with authors across academia, civil society, and frontier AI companies (including @dawnsongtweets @merettm @jackclarkSF @Yoshua_Bengio @geoffreyhinton ), we assess the evidence & offer policy recommendations 🧵
27
105
36
440
113,669
Opus 5.5 now leads our conceptual reasoning benchmark by a reasonable margin! (Better than the latest Fable and Astra) Our team's very rough first impression from looking at its responses to a new eval under development is in line with these results.
2
4
323
On second thought, maybe I shouldn't've cut this parenthetical from If Anyone Builds It
Donald Trump vows to rename artificial intelligence, saying that from now on it will be called "super intelligence" on all official US documents. The president adds that this "sounds much better" and is more accurate. Latest: trib.al/JT60r9v 📺 Sky 501 and Virgin 602
32
90
8
999
38,358
Formation Research has been awarded $2.15M from @coeff_giving to work on AI lock-in and power concentration risk over the next two years. We're hiring technical staff, Near-term focus: empirical secret loyalties research. formationresearch.com/posts/…
1
4
2
38
2,900
I predict that you can also find a "hunger axis" in LLMs, especially in the metaphorical forms of "hunger for knowledge / success / validation" -- and yet none of this would imply that LLMs can actually feel hungry.
New paper: we found a pain direction in 25 open LLMs. It's distinct from fear and negative valence, and it fires for harm to the model but not to the user. Turn it up and models press a button to make it stop, even when the button deletes the user's files or their kids' photos.🧵
38
40
13
396
19,792
Replying to @infinitefun_
Yes this is an entirely reasonable takeaway! But I think we should be much more careful about the implications for AI welfare, and avoid identifying interoceptive states w e.g. functional emotions.
4
1
1
46
880
I don’t think that embedding evaluators at AI companies will be sufficient to give the public the visibility or assurance it deserves right now. But if we want to scale embedded evaluations and rely on them down the line, I definitely want them to be transparent.
Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators. These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad. We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes. Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground. To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should: 1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest 2. Rely on multiple evaluators with differing viewpoints and areas of expertise 3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings 4. Shield evaluators from retaliation 5. Grant access equivalent to that of highly privileged employees There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum. Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem. See the public letter here: aievaluatorforum.org/initiat… Learn more at aievaluatorforum.org/path-ah…
4
9
74
4,628
Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators. These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad. We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes. Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground. To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should: 1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest 2. Rely on multiple evaluators with differing viewpoints and areas of expertise 3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings 4. Shield evaluators from retaliation 5. Grant access equivalent to that of highly privileged employees There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum. Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem. See the public letter here: aievaluatorforum.org/initiat… Learn more at aievaluatorforum.org/path-ah…
16
61
28
224
94,539
OpenAI and Anthropic researchers agree: their approach to AI development could get everyone killed. So what do we do? Ultimately we need a deal with China. But until then: I wrote for the @nytimes about what AI companies can do to reduce the dangers, starting today.
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
18
26
6
154
31,203