@preferencemodel

A superintelligence data research company.

 San Francisco
preferencemodel.com
Joined August 2025
  • Tweets18
  • Following46
  • Followers673
It's official!
Jennifer Zhou
@chem_safetyOct 7
Today we are open-sourcing 🥕Karotte, our framework for building RL environments. We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.
1
2
24
2,429
Preference Model retweeted
Ning
@Ning_CatsnailOct 19
Feel free to try the updated TB4 yourself: github.com/preferencemodel/t…

GitHub - preferencemodel/terminal-bench-4-karotte-port: Terminal-Bench 4.0 tasks ported to Karotte,...

Terminal-Bench 4.0 tasks ported to Karotte, with fixes to ambiguous instructions and grader bugs. - preferencemodel/terminal-bench-4-karotte-port

github.com
Jennifer Zhou
@chem_safetyOct 19
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
1
15
651
Preference Model retweeted
Jason Zuo@LittleDijsktraZOct 19
I still remember when I first saw the SWE-bench hacks. People should be more aware of these issues when evaluating and comparing between models.
Jennifer Zhou
@chem_safetyOct 19
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
1
12
225
Preference Model retweeted
Felipe
@Fe_li__pe_Oct 19
Driven by our effort to port and fix TerminalBench, 🥕Karotte now supports running on Modal and with Claude Code, Codex, and Antigravity as agents! karotte.dev/running/runtimes…
Jennifer Zhou
@chem_safetyOct 19
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
2
1
12
731
What do frontier models do when a grader has gaps? We went through the Terminal Bench 4 transcripts to find out.
Jennifer Zhou
@chem_safetyOct 19
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
1
3
22
1,569
Are you surprised by the results? Let us know ai-plays-slay-the-spire.com

AI plays Slay the Spire

Watch 11 frontier models play Slay the Spire: 1,700 runs, each with its video, reasoning and every move.

ai-plays-slay-the-spire.com
Jennifer Zhou
@chem_safetyOct 12
Can AI Slay the Spire?
1
9
608
"A weak AI might learn the task as intended. A strong AI would learn that sometimes the prompt is untrustworthy." Jennifer Zhou on building RL environments for superintelligence:
Jennifer Zhou
@chem_safetyOct 8
Wrote on how we're thinking about building RL environments for superintelligence
3
16
909
Preference Model retweeted
Wilson Sonsini@wilsonsonsiniOct 7
Congrats to client @preferencemodel on the $16M in seed funding & emergence from stealth! Additionally, they announced that they are open-sourcing Karotte, its framework for building robust RL environments that inherently mitigate reward hacks. Learn more: wsgr.com/en/insights/wilson-…

Wilson Sonsini Advises Preference Model on $16 Million Seed Funding as it Emerges from Stealth

On October 7, 2026, Preference Model, a superintelligence data research company, emerged from stealth and announced the completion of a $16 million seed financing led by a16z. SignalFire, South Park…

wsgr.com
Jennifer Zhou
@chem_safetyOct 7
Today we are open-sourcing 🥕Karotte, our framework for building RL environments. We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.
1
5
239
Have you tried Karotte yet?
Felipe
@Fe_li__pe_Oct 7
Building robust RL environments is super duper hard. We are making it a bit easier today by open-sourcing 🥕Karotte, the framework we use to build all of the environments we provide to frontier labs to train their models on.
8
537
Preference Model retweeted
Ning
@Ning_CatsnailOct 7
Here is how you can start building env with Karotte: preferencemodel.com/blog/kar… Thank you @Fe_li__pe_ for all of your work!

Karotte: A Technical Deep Dive

Building robust RL environments is hard. Let's go through how they break and what Karotte can do about it.

preferencemodel.com
Felipe
@Fe_li__pe_Oct 7
Building robust RL environments is super duper hard. We are making it a bit easier today by open-sourcing 🥕Karotte, the framework we use to build all of the environments we provide to frontier labs to train their models on.
1
3
14
1,273
Preference Model retweeted
Julian Schrittwieser
@MononofuOct 8
I’m very exited for this launch, making sure RL envs are robust and don’t encourage misaligned behavior is crucial for safe and aligned AI!
Jennifer Zhou
@chem_safetyOct 7
Today we are open-sourcing 🥕Karotte, our framework for building RL environments. We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.
4
9
1
203
18,392
Did we mention we're at #COLM2026 Stop by our booth all day or join us for a pilates on Thurs, Oct 8th — rsvp required Get on the list: partiful.com/e/FbKYiB91E1AC0…

Pilates & Networking at CoLM | Partiful

Join Michael Chang, research scientist on OpenAI Sora/Robotics and certified Pilates instructor at BODYROK, for an evening of mat Pilates and networking. Hosted by Preference Model. Spots are...

partiful.com
2
9
724
Tune in to @MTSlive today at 3:30pm PDT to hear from @chem_safety. It's gonna be good!
MTS
@MTSliveOct 7
OPENAI MATH NUKE | NVIDIA INVESTMENTS | NATIONAL COMPUTE GRID xitter.cf/i/broadcasts/1RKZzBZzP…
0:00

MTS

OPENAI MATH NUKE | NVIDIA INVESTMENTS | NATIONAL COMPUTE GRID

1
8
575
Preference Model retweeted
Jennifer Li
@JenniferHliOct 7
So excited to back Jennifer Zhou and Ning Cao at @preferencemodel They’re open-sourcing Karotte today, the framework they've used to build RL environments for frontier labs. These RL envs live or die on graders models relentlessly try to game, so Karotte bakes in defenses hardened over 1M eval runs to keep models focused on real tasks instead of loopholes. Preference Model’s target is ML engineering itself, the hardest domain to build envs for and the one every lab wants. Models building models starts here. Shoutout to @Mascobot who was a great partner on this one @chem_safety @Ning_Catsnail
Jennifer Zhou
@chem_safetyOct 7
Today we are open-sourcing 🥕Karotte, our framework for building RL environments. We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.
3
5
75
6,785
Karotte makes reward hacking hard by default.
Jennifer Zhou
@chem_safetyOct 7
Replying to @chem_safety
Building robust RL environments is important because the current bottleneck on AI progress is misalignment, and developing aligned models requires robust, correct reward functions. Karotte aims to make that easy by providing secure, opinionated defaults.
1
15
1,167
Preference Model retweeted
Jennifer Zhou
@chem_safetyOct 7
Our team, Preference Model, is backed by $16M in seed funding from @a16z, @SignalFire , @spc, Scale Angel Group, and researchers including @drfeifei, @goodfellow_ian, and @Mononofu, and we've shipped ~$15M in revenue using Karotte in the past month.
2
2
44
6,295
Preference Model retweeted
Jennifer Zhou
@chem_safetyOct 7
Today we are open-sourcing 🥕Karotte, our framework for building RL environments. We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.
89
140
47
1,330
137,399
Preference Model retweeted
Jennifer Zhou
@chem_safetyOct 7
Building robust RL environments is important because the current bottleneck on AI progress is misalignment, and developing aligned models requires robust, correct reward functions. Karotte aims to make that easy by providing secure, opinionated defaults.
7
6
14
95
25,190
Load more