Evals, RL environments & AI data | Building benchmarks for binary reasoning.
Austin, TX
Joined December 2009
- Tweets51.1K
- Following1.4K
- Followers1.4K
- Likes6.8K
Marq retweeted
you either die a
Sent from my iPhone
or you live long enough to see yourself
Authored by Claude Code
Marq retweeted
Marvel Studios has confirmed that Patrick Stewart's final on-screen performance will be as Professor X in ‘AVENGERS: DOOMSDAY’.
Marq retweeted
"real envs > synthetic envs" feels like mostly 3 things:
- a complaint about sloppy synthetic data pipelines
- a way to make envs pricier without making them better
- synthetic task authors guessing the task distribution because they've never worked with real production traffic at scale
real envs ship very hackable tasks too. whereas a carefully built synthetic env that's realistic enough has all the same benefits as real envs, + you get to aim at one failure mode on purpose and isolate it from the random noise
😂
Qwen 3.8 27B is actually the most brilliant retarded model that I have ever used. 😅
It spends about 2/4 of its reasoning trace coming up with the most retarded ideas known to man, and then an additional 1/4 convincing itself that its ideas are completely retarded.
It then manages, from the ashes of retardation, to arise towards the cleanest, most efficient, and bug-free implementation of any open-weight model of this size.
It kinda works like American politics, in a way?!
Marq retweeted
This is crazier than any spy movie:
According to The New York Times, a Chinese businesswoman with close ties to Nvidia CEO Jensen Huang was passing intelligence to Beijing, and Trump was briefed but took no action. What she shared remains unclear, and U.S. officials initially believed Huang was unaware of her alleged intelligence role.
Breaking News: A Chinese businesswoman who had a close relationship with Nvidia’s CEO, Jensen Huang, was found to be passing intelligence to Beijing. nyti.ms/3Ukob1t
Marq retweeted
how do you post-train an agent?
we laid it out as a 5-step curriculum, from SFT to RL through a real harness, with livestreams, code and docs for each step
huggingface.co/blog/huggingf…
I can do one better I was rejected from the CIA after going through their test. Cyber security position 😵💫
I just got rejected in the final round for AI Researcher at @Anthropic.
I applied with 4 months of experience. Here are my takeaways 🧵
Marq retweeted
Gemini 4 Carbon is our next frontier model.
It’s state-of-the-art across cybersecurity and long-horizon capabilities.
Marq retweeted
A few thoughts on Anthropic’s approach:
1. Drug discovery will be one of the most significant revenue streams in the future; the goal here is to set the right course early on. Most diseases will become treatable, a market encompassing 7 billion people.
2. The timing is clearly intentional, announced just before the IPO to signal to investors that the business will continue to thrive in the future.
3. This will likely make it clear, even to those outside the "tech nerd" sphere, why AI is so important. It goes beyond just writing code; it solves problems for humanity.
One of the best people to follow
i've found defects in ~65% of the TAU-banking tasks.
these are tasks where the policy isn't enforced correctly. for example, if there are rules the model can violate and not get penalized for (transferring to human reqs, tasks where the tool call can fail and the model is still rewarded, etc). some grading may be directionally correct, but still contains noticeable defects that needs to be hardened prior to use.
high-level defects for people working on benchmarks right now:
> models are capable of taking many out of scope actions without being penalized (only tool-call checks, no db)
> models can directly lie to the customer with no penalty (no communication checks, which for a customer service benchmark feels important, though i didn't label tasks that only fail this as p0)
> many of the tool call checks are naive and if the tool call fails there are no repercussions. i.e. a test of email change may just check if the tool was called, with no field checks or follow through check, so if the tool call fails or if i change the wrong email it may still pass.
to expound a bit more:
(1) i'm going to try and go through every public benchmark. the reason for these audits is that i've become frustrated with people using defective benchmarks to choose what type of data they want to purchase and data co's making flawed data to help a model hill-climb a benchmark.
it is very clear that bad RL data is a contributor to reward hacking, and the benchmarks that inform purchasing decisions lie upstream of that, thus handing a huge power to companies that publish benchmarks.
(2) sierra and the TAU series have had tremendous contributions to tool-calling evals. tau-bench is actually what got me into RL data in the first place more than two years ago. this is in no means to discredit sierra and the effort they've put into creating the benchmark. big inspiration!
(3) please reach out and correct me if i'm wrong on any of these asap! i want to give sierra credit where it is due.
Marq retweeted
Axiom says its AI agents formalized a 15,000-page mathematical proof in just 24 days, largest proofs in mathematics.
Built by over 100 mathematicians across decades, the proof classifies the fundamental building blocks of finite symmetry.
With human guidance, the agents translated it into Lean, making every logical step computer-checkable and repairing errors along the way.
AI tackling an entire interconnected body of mathematics, with results others can verify. A major milestone.
We are witnessing history unfold. Again. Every day now.
