@lusxvr

ELLIS PhD Student @ TUM

Joined January 2015
  • Tweets272
  • Following528
  • Followers1.1K
41Media
Pinned Tweet
Luis
@lusxvrOct 6
Agents are evaluated with ever more detailed metrics, but why don't we study how their own configuration shapes performance? We study this on 4 scientific tasks, by varying 5 parts of the agents configuration across 18,000+ runs. Here are my favorite findings:
1
4
14
648
Paper Spotlight: What if we studied AI the way biologists study animals? In 2019 (at the time of GPT-2!), researchers argued in Nature for a new science: "machine behaviour". Today, it looks like a pretty good prediction of the current scientific field around frontier systems:
2
5
22
656
more replies
In 2019, studying a machine/algo was hard, especially outside CS. Language interaction was not possible yet and most methods and algorithms were domain specific systems.
2
18
Today anyone with an API key can run a behavioural experiment on a model in an afternoon, on models rivaling or surpassing humans in some areas. The bottleneck moved from access to asking good questions, and the stakes are growing with the capabilities.
1
18
Paper Spotlight: What does an AI agent do when nobody gives it a task? The authors gave Claude Code a body on an unknown island, said only "Begin", and left it alone for 30 hours. It climbed hills, built towers, invented basketball and bowling, and ran physics experiments.
3
4
1
29
1,225
more replies
You study what the finished system does, the way a psychologist or an ethologist would study an animal. That brings AI closer to psychology, philosophy, cognitive science and the social sciences, which have studied similar questions for decades.
1
1
67
We as computer scientists, therefore unlock a whole exiting new world of possible directions in which we can study modern AI, largely limited by our own creativity to come up with settings to study and questions to ask.
2
58
Paper Highlight: Towards a Science of AI Agent Reliability Agents keep scoring higher on benchmarks, yet they still fail in real deployments. Accuracy alone hides this, and agent reliability barely improved over two years of model releases.
7
2
1
10
440
more replies
If agents tackle more long-running and hard-to-verify tasks, their behavior in these dimensions will grow in importance, and should therefore also gain bigger importance throughout their development and evaluations.
1
31
Overall very interesting and important work by @steverab @sayashk @PKirgis Kangheng Liu @SaitejaUtpala @random_walker ! Paper: arxiv.org/pdf/2602.16666
63
Load more