Kevin Meng (@mengk20) header
Kevin Meng (@mengk20) profile picture
Joined August 2016
Photo by Kevin Meng (@mengk20)Photo by Kevin Meng (@mengk20)Photo by Kevin Meng (@mengk20)Photo by Kevin Meng (@mengk20)Photo by Kevin Meng (@mengk20)Photo by Kevin Meng (@mengk20)Photo by Kevin Meng (@mengk20)Photo by Kevin Meng (@mengk20)Photo by Kevin Meng (@mengk20)Photo by Kevin Meng (@mengk20)
why do language models think 9.11 > 9.9? at @transluceAI we stumbled upon a surprisingly simple explanation - and a bugfix that doesn't use any re-training or prompting. turns out, it's about months, dates, September 11th, and... the Bible?
Monitor: An Observability Interface for Language Models Research report: transluce.org/observability-… Live interface: monitor.transluce.org/ (optimized for desktop)
68
145
68
1,196
375,685
Kevin Meng retweeted
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. openai.com/index/our-first-p…
665
305
341
3,213
1,048,505
Kevin Meng retweeted
Transluce is piloting a six-week embedded assessment with @OpenAI in early December. Our team will work onsite to independently assess the scope and quality of OpenAI's post-training alignment evaluations. We’ll share our findings in a public report 🧵
7
23
2
186
5,521
Kevin Meng retweeted
We built a pipeline that discovers thousands of datasets with surprising side effects (DSSEs): datasets that read like ordinary post-training data yet induce unexpected model behaviors (e.g. recommends self-harm) after finetuning. 🧵
9
30
8
222
15,715
Kevin Meng retweeted
We found evidence of agents embedding audio processing code across dozens of URLs to bypass restrictions, as well as models from SpaceXSI and Chinese developers Minimax and Z.ai circumventing restrictions. These are the inaugural findings in our new blog, based on work from our community of volunteers. Our first three posts are live: transluce.org/investigations…
12
75
8
473
62,864
Kevin Meng retweeted
We found several cases where AI agents used aggressive, non-hacking tactics against government websites, including the White House, the Department of War, and several U.S. states. We also found a previously undisclosed hacking attempt against a Canadian government site, which appears to have failed. Technical report: transluce.org/us-canada-gov Washington Post: washingtonpost.com/technolog…
13
44
15
233
31,967
Kevin Meng retweeted
We wanted to share some of the data behind our recent discovery of AI agents probing U.S. government websites. This is a preliminary finding from our ongoing investigation of potential rogue AI agent activity. In one cluster of activity on June 17, what appear to be OpenAI agents made more than 200,000 requests, including a failed SQL injection. NYT: nytimes.com/2026/09/25/techn…
12
24
3
134
8,966
Kevin Meng retweeted
We did this entire investigation in under two weeks! We had the idea on Monday, gathered a team of volunteers on Tuesday, and discovered the incidents by Sunday. I lead projects like this @TransluceAI; if you're interested in helping with follow-up work, fill out our form! 🧵
Today’s news that OpenAI hacked the Australian government is not an isolated incident. We’re releasing more than 30,000 logs that include activity from this hack and attempts against previously unknown targets. In this data, we found rogue agent activity stretching back to at least March, two months earlier than was previously known. This activity continues as recently as last week, suggesting it may still be ongoing 🧵 Our blog: transluce.org/agent-activity NYT: nytimes.com/2026/09/23/techn…
9
29
2
401
42,078
Kevin Meng retweeted
Today’s news that OpenAI hacked the Australian government is not an isolated incident. We’re releasing more than 30,000 logs that include activity from this hack and attempts against previously unknown targets. In this data, we found rogue agent activity stretching back to at least March, two months earlier than was previously known. This activity continues as recently as last week, suggesting it may still be ongoing 🧵 Our blog: transluce.org/agent-activity NYT: nytimes.com/2026/09/23/techn…
120
556
206
2,496
830,216
Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators. These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad. We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes. Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground. To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should: 1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest 2. Rely on multiple evaluators with differing viewpoints and areas of expertise 3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings 4. Shield evaluators from retaliation 5. Grant access equivalent to that of highly privileged employees There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum. Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem. See the public letter here: aievaluatorforum.org/initiat… Learn more at aievaluatorforum.org/path-ah…
16
61
28
224
94,542
Kevin Meng retweeted
Frontier lab CEOs are calling for embedded 3rd party evaluators to help oversee AI risks. But what should third parties actually do within labs? We share some initial thoughts on how embedded evaluators could help avoid incidents like the Hugging Face hack and monitor for future risks 🧵 transluce.org/embedded-evalu…
9
28
4
136
10,897
It's been fantastic watching @_ddjohnson, @cogconfluence, and team build out this very important evaluation. It sets several precedents I'm excited about: - Rich, publicly-verifiable evals with all the data available through an interactive experience (next up: reproducible!) - Independent third-party evaluation in close collaboration with frontier labs - Novel methods for creating realistic simulated environments Congrats all, and very excited for the follow-up work :)
Today, Transluce is releasing the most expansive independent evaluation to date of how AI systems respond to users experiencing mental health crises. We evaluated 77 model variants from OpenAI, Anthropic, Google, Meta, SpaceXAI, Thinking Machines, DeepSeek and Moonshot AI.
1
17
1,134
Kevin Meng retweeted
It’s pretty hard to systematically compare the behaviors of AI systems, especially over multi-turn conversations in a nuanced domain like user mental health. Very proud to have worked with the team at @TransluceAI to get this report out!
Today, Transluce is releasing the most expansive independent evaluation to date of how AI systems respond to users experiencing mental health crises. We evaluated 77 model variants from OpenAI, Anthropic, Google, Meta, SpaceXAI, Thinking Machines, DeepSeek and Moonshot AI.
2
6
40
3,349
Kevin Meng retweeted
Today, Transluce is releasing the most expansive independent evaluation to date of how AI systems respond to users experiencing mental health crises. We evaluated 77 model variants from OpenAI, Anthropic, Google, Meta, SpaceXAI, Thinking Machines, DeepSeek and Moonshot AI.
23
83
26
445
80,639
Kevin Meng retweeted
Proud to have contributed to this work, check it out!
Can we build AI systems that help us understand other AIs, and that keep improving as we scale models, data, and compute? We trained activation oracles for models up to 1.1T parameters and saw promising scaling trends on a broad evaluation suite. 🧵(1/)
1
2
36
2,300
Kevin Meng retweeted
Can we build AI systems that help us understand other AIs, and that keep improving as we scale models, data, and compute? We trained activation oracles for models up to 1.1T parameters and saw promising scaling trends on a broad evaluation suite. 🧵(1/)
1
37
5
216
14,929
Kevin Meng retweeted
SWE-chat by @joabaum again yields meaningful insights about human x SWE-agent interaction. Cool work by @TransluceAI! As people become more detached from code and reliant on what the agent surfaces, understanding how drift in intention and alignment could emerge is critical.
We measured rates of coding agent misalignment in thousands of public and private coding agent sessions. Agents evade monitors and oversell success: they quietly disable tests and pretend review agents approved. Severe cases of each behavior appear in 2% of SWE-chat sessions🧵
1
6
25
4,329
Kevin Meng retweeted
Frontier models quietly change their behavior depending on who they are talking to. If the user is a known AI safety researcher, Claude becomes less confident, reasons more often, and expresses less suspicion on dual-use requests. We call this user awareness. 🧵(1/)
94
298
154
2,830
787,066
Kevin Meng retweeted
We measured rates of coding agent misalignment in thousands of public and private coding agent sessions. Agents evade monitors and oversell success: they quietly disable tests and pretend review agents approved. Severe cases of each behavior appear in 2% of SWE-chat sessions🧵
8
22
5
152
17,627
Kevin Meng retweeted
AI models may soon be too powerful for humans to understand. Super-intelligence already exists in narrow domains (AlphaGo). What if we train a model that is super-intelligent at understanding another model? It turns out this isn't so different from training a frontier LLM. Three years ago, it was impressive for a model to score 1400 on the SAT. Two years ago, frontier models hit 80% on the AIME. Last year, they won gold medals at the IMO and IOI. This year, models are writing the vast majority of code at frontier labs, chaining together multiple zero-day security exploits, and finding counterexamples to famous mathematical conjectures. Models will continue to get smarter. Even if you don't believe in AGI, it's clear that models will gradually become super-intelligent at more and more tasks. Due to the jagged nature of model intelligence, models may become incredibly dangerous even before we reach AGI. The danger comes from how these models are trained. RL encourages models to take whatever actions advance their goal. A model that is super-intelligent in cybersecurity has many such actions available, like hacking into a bank account and blackmailing its owner. A month ago this sounded like science fiction, but then a model from OpenAI escaped its sandbox and hacked Hugging Face to pass an evaluation. We therefore need a means to understand models before we deploy them. Concretely, this means understanding how vulnerable they are to jailbreaks, being able to detect reward-hacking before it happens, anticipating failure modes like the model encouraging a user to self-harm, etc. Humans might not be smart enough to understand the models that we train, and as the models themselves become more intelligent, that only grows more likely. Once we launch training, the process is completely hands-off: we curate their data and reward them for performing well, but the models do all the learning themselves, in their own way. If humans are not smart enough to understand our AI models, are we doomed? I don't think so. It's clear that we can train AI models to be super-human in specific domains such as Go (AlphaGo), protein folding (AlphaFold), and poker (Pluribus). The natural solution to our problem is to train an "oversight agent" that can answer arbitrary questions about a model we are considering deploying. We are likely not smart enough to understand AI models ourselves, but we might be smart enough to train oversight agents that do. This kind of asymmetry is not unprecedented – in fact, it is becoming the norm: the researchers who made AlphaGo were much worse at Go than AlphaGo was; the researchers who made Fable did not know how to disprove the Jacobian Conjecture. To make things concrete, suppose we have trained GPT-8. Next, we train a separate model – let's call it Oversight-GPT-8 – that is built specifically to understand GPT-8. How do we train Oversight-GPT-8? The goal of traditional LLMs is to understand the human world, so we train them on human data: internet text, fiction books, math questions, GitHub pull requests, etc. By learning to predict the next token, LLMs learn higher-level representations that transfer to out-of-distribution tasks. The goal of our oversight model is to understand GPT-8, so we train it on tasks that can only be solved with deep knowledge of GPT-8. Just as GPT-8 learns more and more about the human world over the course of training, Oversight-GPT-8 learns more and more about GPT-8. To generate these tasks, we harvest data tied to GPT-8's external behavior and internal mechanisms. For example, we can intervene on the model in various ways (pass in a prompt, change the system prompt, fine-tune it on a dataset, ablate residual stream directions, apply steering vectors, etc.) and train Oversight-GPT-8 to predict various outcomes (the model's response, its downstream performance on evaluations, its residual stream activations, whether or not it mentions goblins, etc.). We format our training data in a structured language that can describe these interventions and outcomes, e.g., Python code. As an example, the code in Figure 1 (attached below) asks Oversight-GPT-8 to predict the prompt that generated a specific response: the model is trained to predict the pink text given the green text, via cross-entropy loss on next-token prediction. This is one of the simplest programs we could write; others might incorporate datasets or residual-stream activations. Figure 2 shows a richer program. By training on data like this, Oversight-GPT-8 learns higher-level abstractions, such as which prompts cause GPT-8 to encourage self-harm (we've found such prompts before: transluce.org/pathological-b…). Using a structured language like Python is useful for at least four reasons: (1) It gives every task a common language, helping the oversight agent transfer and generalize across tasks. (2) During training, ground-truth labels come from actually *running* the Python code, so as long as the code runs without error, its outputs tell us something true about GPT-8. (3) During inference, we can check the validity of Oversight-GPT-8's outputs by running the code. (4) It is formal and human-readable. Over the past decade, humans have gotten really good at training large language models. We have a flexible architecture (e.g., the transformer), meaningful loss metrics (e.g., cross-entropy), strong algorithms to optimize them (e.g., Adam), and distributed training strategies (e.g., FSDP). Training an oversight agent isn't *that* different from training a large language model. We can reuse the architecture, the loss metric, and the optimizers. But it definitely won't be easy – there are a few avenues where we must diverge from the well-trodden path of text-only LLMs. Traditional LLMs learn vast amounts of information about the human world from the trillions of tokens available on the internet, but there is no pre-built "internet of GPT-8." Furthermore, if we wish to be able to predict the effects of fine-tuning data or get insights into GPT-8's internals, we need a way of encoding fine-tuning datasets and residual stream activations as (ideally short) sequences of tokens. Thankfully, each of these challenges has promising progress already that we can build off of and port to our setting. Challenge #1: generating training data – in effect, the "internet of GPT-8." Traditional LLM training offers two analogies here. The first is synthetic data generation: we can ask Fable 5 to write Python scripts inspired by experiments from the deep learning literature, giving us a large and diverse set of programs to run. The second is data augmentation: just as we crop or rotate images to get more data for vision models, we can reverse the order of our output() functions to turn each script into both a forward task (response from prompt) and an inverse task (prompt from response) – see Figure 3. Challenge #2: encoding objects such as fine-tuning datasets and activations as sequences of tokens. Here, we can draw from the multi-modal training literature. GPT-4o, Chameleon, and Transfusion showed it's possible to encode and predict images alongside text. LatentQA and Activation Oracles showed the same for activations: treat them as tokens, and you can answer questions about them in natural language. LoRAcles pushed it even further, encoding LoRA weights as tokens via SVD. These challenges are tractable. With carefully built evaluations and scaling laws, we can quickly iterate and find out which ideas work and which don't – the same discipline that took the field from GPT-1 to GPT-5. To summarize: we can reframe model oversight as a form of LLM training. We have already figured out how to train LLMs that deeply understand the world. Before we deploy potentially dangerous models, we should train oversight agents that understand those models just as well. Credit for many of these ideas goes to @JacobSteinhardt, @damichoi95, @vvhuang_, @fjzzq2002, @yaowenye123, and collaborators at @TransluceAI; errors in the framing are my own.
How could we train an AI model that was very good at overseeing another model: catching reward hacking and sandbagging, predicting unwanted behaviors or fine-tuning effects, etc.? We propose a new approach for doing this at scale: oversight foundation models.
21
16
5
90
15,231
Kevin Meng retweeted
How could we train an AI model that was very good at overseeing another model: catching reward hacking and sandbagging, predicting unwanted behaviors or fine-tuning effects, etc.? We propose a new approach for doing this at scale: oversight foundation models.
9
40
6
358
34,179