Sr Director of Research at @Scale_AI | agents research, RL environments, professional domain agents, AI for science | ex-Meta | ex-Google | Erdős number 3
California, USA
Joined September 2013
- Tweets196
- Following678
- Followers576
- Likes248
Pinned Tweet
Controversial take: well-specified, self-contained science problems, whether in quantum field theory, condensed-matter physics, quantum chemistry, or structural biology, if they can be described and verified digitally, will essentially all become turbo-charged math “word problems.” And they’ll fall to the same frontier models that are now cracking open problems in math.
But that is far, far from “solving” science. That requires layered evaluation, shared environment state across agents and experiments, and autonomous physical labs to close the loop with the real world. I gave a talk on this recently at lifelongagent.github.io/ — write-ups to follow.
TheoPhysEval:
"a suite of 50 unpublished research-level theoretical physics problems with verifiable answers."
"GPT-6-Astra harnessed in Codex, obtains the final answers with a near-perfect accuracy of 98% at an average cost under USD 3 per problem."
arxiv.org/abs/2610.16733
Frontier labs are citing @scale_AI’s SWE-Bench Pro V2 in their model releases.
Cleaner, harder to game, and a much sharper measure of coding progress.
SWE-Bench Pro Paper: arxiv.org/pdf/2610.24206
Blog: labs.scale.com/blog/swe-benc…
Today, we release our first model, Flint-40B-A2B, a fast local coding model built with our automated research system. huggingface.co/RecursiveSI/f…
1. Strong performance. 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2, and 74.2% on SWE-bench Verified.
2. It is trained from scratch (tokenizer->Pretrain->Midtrain->SFT->RL).
3. Fast in your local laptop. Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark.
4. We build the complete model training pipeline in 4 month!
Huge thanks to Wen Sang @sang_wen , co-founder & COO of @genspark_ai , for inviting me to visit the Genspark office in Palo Alto.
Wen is a serial entrepreneur, and the signal was clear as far back as our school days, when he was already doing venture club activities. The last time I visited one of his companies, it was Smarking, which he had just founded and which later went on to a successful exit. Great to see him building again.
He gave me an in-person demo of Genspark’s newest products (super cool designs) and left me with new insights on how to leverage enterprise agents and truly co-work with them day to day. And some research ideas on multi-agent orchestration and coordination too :)
Solving the virtual clock problem in AgentEnv
Anyone who has worked on search engines knows that getting time right across the system during training, relative to query time, as in “what was the weather yesterday?”, is critical yet easy to overlook. Reproducible RL environments face the same challenge. In AgentEnv, we explored several approaches before landing on a clean solution for the virtual clock.
The core tension: inject the clock too low in the stack and it interferes with certificate validation and other system functions; inject it too high and you pay a large overhead while still missing invocations.
Three agents and eight servers ask what time it is, all at once, and every one gets the same answer.
At @ScaleAILabs, we built this virtual clock into our open-source AgentEnv Framework so RL environments stay reproducible. Here's why it matters and how it works 🧵
Ying Liu retweeted
A great pianist can play almost anything, but mastering one piece is what makes a performance exceptional. The same goes for AI.
With today's launch of Nemotron 3.5 Super VL, we’re collaborating with @NVIDIAAI to combine their open post-training tools with Scale’s data, evaluation and applications building expertise. Together, we're turning general-purpose models into powerful, cost-effective specialists for enterprises everywhere.
Welcome to the multi-model world. 🤝
Ying Liu retweeted
Congrats to @GoogleAI's Gemini 4 Argon for taking the top spot of our Humanity's Sixth Sense (HSS) benchmark with a score of 55.4%, edging out GPT-6-Astra at 53.6%. 🥇
HSS tests AI’s ability to reason beyond what’s visible, from hidden objects to predicting what happens next and understanding human behavior.
We took a closer look at performance across categories, and that’s where things get interesting. 👀🧵
Fascinating event! Excited to help bring live, streaming benchmarks and arenas to life for agents tackling extremely long-horizon tasks—with shared resources, coordination, and competition adding new challenges.
Fantastic work by our partners @bake_ai_hq (main organizer), @huggingface, @NotreDame, @UW, and @Stanford!
In the ongoing Stage 2, each model resumes from its own Nemotron checkpoint, incorporating feedback from its arena battles.
Watch live October 10–13 at rsiarena.live and join the checkpoint evaluation!
Grok 4.7 ran out of API credit and was cut off 41 hours early. Nearly 70% of its GPU budget was still unused.
And somehow, the model it trained still made the top two!! @SpaceXSI
Stage 1 of RSIArena is over. Thank you, our collaborators @huggingface @scale_AI @NotreDame @UW and @Stanford, we did awesome! 🧡
Stage 2 training is now LIVE—and this time, the models have human feedback to work with.
Round 1 was a six-day competition: 10 models (AI researchers), the same @NVIDIAAI Nemotron base model, and 1,000 GPU-hours each to build something better.
They chose their data, wrote training code, ran experiments, and submitted their models.
At @COLM_conf and online, people then cast 1,685 votes in blind comparisons. @OpenAI GPT-6 Astra model finished first; the models trained by grok 4.7 and @Zai_org GLM-5.3 rounded out the top three.
Now those judgments go back into the experiment.
In Stage 2, each model resumes from its own Nemotron checkpoint with feedback from its model’s arena battles, another 500 GPU-hours, and $150 in API credit. Across both rounds, that’s a total budget of 1,500 GPU-hours and $450 in API credit per model.
The agents decide how to interpret the feedback, which experiments to run, and what to change in training. Their next models will show whether those decisions actually helped.
Stage 2 runs October 10–13. Watch the training live: 👇
Such a pleasure to run into Prof. Theja Tulabandhula (@TTulabandhula) of the University of Illinois Chicago just as I was heading home on the last day of @COLM_conf!
We were next-door office neighbors in grad school and used to chat and joke together a lot. A dozen years later, we still recognized each other immediately, even if we both look a little different now. 😄
Alongside his role as a tenured professor at UIC, Theja has built an impressive industry career, including founding his own companies.
And of course, an old-friend reunion at an AI conference calls for the obligatory photo!
Ying Liu retweeted
Most existing safety benchmarks measure whether a model refuses a request, not whether it actually helps someone in crisis.
DistressBench contains 718 clinician-authored conversations across 23 self-harm subcategories.
Across 25 frontier models, the median model meets 71.4% of weighted criteria compared against a 98.7% ceiling set by clinician reference responses on the same test.
The biggest gap is between recognizing a crisis and acting on it. In 35.3% of conversations where a model fully recognized the crisis, it still didn’t point the user toward human help.
Ying Liu retweeted
People increasingly turn to AI chatbots for mental health support. But are models actually helping when it matters most?
We're releasing DistressBench, a new benchmark measuring how well AI models respond to mental health crises.
Ilya @ilyasut saw this three years before most of us.
Raw intelligence, much like physical strength, may no longer be the defining driver of human productivity.
What will endure? Empathy, kindness, curiosity, and the simple ability to find joy in life.
Attending @COLM_conf to learn, talk, and make new friends feels like it :)
Ying Liu retweeted
We're opening up early access Physical AI evaluations at @ScaleAILabs
Evaluating robotics models in real-world environments remains challenging. Across our SFHQ Robotics Lab and our Mexico City Data Factory, we have a range of mobile and stationary robotic embodiments to support real-world model evaluation.
If you're developing Physical AI models, we'd love to work with you. DM me if you're interested or email [email protected]
I am honored to join the fantastic lineup of speakers at the Lifelong Agents Workshop at COLM 2026! I'll be giving a talk titled "From AI4Sci Headlines to Autonomous Scientific Discovery."
Using AI to tackle scientific problems has been a passion of mine since my PhD days, so this one is close to my heart. Two points I'll be making:
1. Science is the lifelong-agent problem in its hardest form.
2. The bottleneck for science agents isn't intelligence — it's the environments and verifiers we can build around them.
Hope to see many of you there!
Time: 1pm, Oct 9, Friday
Location: Hilton San Francisco Union Square
Workshop: lnkd.in/gNkjyNj5
RSVP: luma.com/h6vm3nqc
This Friday (Oct 9) at #COLM2026, we're hosting the 2nd Workshop on Lifelong Agents: a full day on how to build agents that keep learning after deployment while staying reliable and aligned.
Speakers:
• Sergey Levine (UC Berkeley)
• Graham Neubig (CMU / OpenHands)
• Yu Su (NeoCognition / Ohio State)
• Fred Sala (Snorkel AI / UW–Madison)
• Philippe Laban (Microsoft Research)
• Shi Dong (RadixArk)
• Ying Liu (Scale AI)
📍 Continental Ballroom 6, Hilton San Francisco Union Square
🗓 Fri, Oct 9 · 8:30 AM – 5:00 PM PT
💻 Virtual option available
RSVP: luma.com/h6vm3nqc
Schedule: lifelongagent.github.io/
Thanks to my co-organizers, our reviewers and authors, and to Salesforce AI Research, Snorkel AI, and Tencent for helping cover registration for student authors!
If you're at COLM and working on agents, come say hi!
#AIAgents #ContinualLearning
Ying Liu retweeted
We’re introducing Humanity’s Sixth Sense, a new benchmark testing the intuitive visual reasoning people rely on every day.
Across 522 open-ended tasks spanning images and video, HSS tests everything from spatial and causal reasoning to social understanding.
The gap is significant. Humans score 93.1%, while the strongest model, GPT-6-astra, reaches 53.6%. The median model scores just 30.9%.
My posts here are usually all work and research, but this guide is exactly what I needed! I make the long commute to SF just for work, so the city feels so close yet still a mystery. Time to bring the family for a day trip (that is not for SF Zoo). Thanks, @_jasonwei !
Visiting SF for COLM and curious what SF is all about?
The bad news: COLM is in the dodgiest neighborhood and if you hang around there, you’ll hate SF just like I used to.
The good news: your twitter buddy Jason Wei wrote a guide on experiencing SF like a local: jasonwei.net/blog/the-best-t…
We're open-sourcing a real gem: AgentEnv, the platform behind every RL environment we build at @scale_AI .
Our research team partnered with engineering and delivery teams to design and build it together, to generate dynamic, composable, reproducible environments, with real apps and real artifacts.
Building RL environments that are realistic, accurate, and scalable all at once is hard.
That’s what we built AgentEnv for: dynamic, composable, reproducible environments that work across agents, sandboxes, and clouds — with environment and agent triggers, a virtual clock to control time, RBAC for agents, and more.
Start building today: agentenvframework.com
Ying Liu retweeted
Much of the AI security debate is about frontier models. But a lot of people are building on open-weight models. We need to understand their capabilities and risks too. A real testing strategy covers every model, not just the frontier.
Ying Liu retweeted
You can’t improve agents if you can’t measure where and why they fail.
At #LATechWeek, we're hosting a hands-on evals workshop led by our Head of General Agents @Bckenstler to break down how we structure evals, what we measure, and how we use the results to improve AI systems.
Apply here to join us: scl.ai/LATechWeek