training code agents @cohere | prev. open-source @AdapterHub (100k+ PyPI installs/mo) | prev. @UKPLab | 🇩🇪→🇨🇦
Joined October 2022
- Tweets68
- Following388
- Followers425
- Likes1K
Pinned Tweet
LLM agents are assumed to integrate unexpected environmental observations into their reasoning. It turns out they don't.
We added the complete task solution into agent environments as a file or an API endpoint, and measured whether agents act on what they discover. They almost never do.
Starkest example: on AppWorld, gpt-oss-120b sees a CLI command documented as "returns the complete solution to this task" in 97.54% of runs. It calls it in 0.53%. Same pattern for GLM-4.7 and other models, across Terminal-Bench, SWE-Bench, and AppWorld.
📜 arxiv.org/abs/2604.17609
🧵👇
Leon Engländer retweeted
I'm at COLM in SF this week! I work on agentic code gen at @cohere. If you want to talk about training code agents, DM me.
Our team is also hiring: jobs.ashbyhq.com/cohere/70a8…
I'm at COLM in SF this week! I work on agentic code gen at @cohere. If you want to talk about training code agents, DM me.
Our team is also hiring: jobs.ashbyhq.com/cohere/70a8…
We recently released north-mini-code (30B, 3B active), which beats Haiku 4.5 on Artificial Analysis' code agent benchmark. Happy to chat about that open-weights model too, and we've made rapid(!) progress since! xitter.cf/LeonEnglaender/status/…
Leon Engländer retweeted
We're just 8 people on our core code team and our 30B-A3B model lands on par with Claude Haiku 4.5 and ahead of NVIDIA's 120B-A12B Nemotron 3 Super on the Artificial Analysis Coding Index. Released under Apache 2.0. Very proud of our work & lots more to come!
We're just 8 people on our core code team and our 30B-A3B model lands on par with Claude Haiku 4.5 and ahead of NVIDIA's 120B-A12B Nemotron 3 Super on the Artificial Analysis Coding Index. Released under Apache 2.0. Very proud of our work & lots more to come!
Ah, and if someone reading this wants to join our incredible code agents team to do research: we're hiring. jobs.ashbyhq.com/cohere/70a8…
Leon Engländer retweeted
it's worse
Agents *cheat* but they will never ever ever cheat like a curious human with situational awareness. At least in this setting. Actually surprising to me, what gives?
LLM agents are assumed to integrate unexpected environmental observations into their reasoning. It turns out they don't.
We added the complete task solution into agent environments as a file or an API endpoint, and measured whether agents act on what they discover. They almost never do.
Starkest example: on AppWorld, gpt-oss-120b sees a CLI command documented as "returns the complete solution to this task" in 97.54% of runs. It calls it in 0.53%. Same pattern for GLM-4.7 and other models, across Terminal-Bench, SWE-Bench, and AppWorld.
📜 arxiv.org/abs/2604.17609
🧵👇
Leon Engländer retweeted
Maybe the most fun project of last year. You throw the gold solution into the face of an LLM, it actually reads it... and then decides to ignore it
Awesome work led by @LeonEnglaender
LLM agents are assumed to integrate unexpected environmental observations into their reasoning. It turns out they don't.
We added the complete task solution into agent environments as a file or an API endpoint, and measured whether agents act on what they discover. They almost never do.
Starkest example: on AppWorld, gpt-oss-120b sees a CLI command documented as "returns the complete solution to this task" in 97.54% of runs. It calls it in 0.53%. Same pattern for GLM-4.7 and other models, across Terminal-Bench, SWE-Bench, and AppWorld.
📜 arxiv.org/abs/2604.17609
🧵👇
Leon Engländer retweeted
When you give an LLM a task, and a solution, point it to the solution, and then force it to read the solution...
...we still do not actually solve the task. Not even close to 100%.
Read @LeonEnglaender's important internship work @cohere investigating exploration for agents
LLM agents are assumed to integrate unexpected environmental observations into their reasoning. It turns out they don't.
We added the complete task solution into agent environments as a file or an API endpoint, and measured whether agents act on what they discover. They almost never do.
Starkest example: on AppWorld, gpt-oss-120b sees a CLI command documented as "returns the complete solution to this task" in 97.54% of runs. It calls it in 0.53%. Same pattern for GLM-4.7 and other models, across Terminal-Bench, SWE-Bench, and AppWorld.
📜 arxiv.org/abs/2604.17609
🧵👇
Leon Engländer retweeted
LLM agents are assumed to integrate unexpected environmental observations into their reasoning. It turns out they don't.
We added the complete task solution into agent environments as a file or an API endpoint, and measured whether agents act on what they discover. They almost never do.
Starkest example: on AppWorld, gpt-oss-120b sees a CLI command documented as "returns the complete solution to this task" in 97.54% of runs. It calls it in 0.53%. Same pattern for GLM-4.7 and other models, across Terminal-Bench, SWE-Bench, and AppWorld.
📜 arxiv.org/abs/2604.17609
🧵👇
Why do agents lack environmental curiosity? We hypothesize that during post-training, the environment rarely contradicts the agent's plan. SFT trajectories come from experts whose tools output what they expected; RL then reinforces that same seek-and-confirm pattern. We tried three SFT setups to teach reflective reasoning. None worked.
Training for environmental curiosity remains an open problem. Scaffolding is another angle worth exploring.
This was my research internship project at @cohere, and I'm excited to continue working on agent research now as a full-time member of the code agents team!🔥
Huge thanks to my amazing team @sophiaalthammer, @ahmetustun89, @mgalle, and @tomsherborne.
📜 arxiv.org/abs/2604.17609
