Chief Research Officer at @Harvey (past: @MicrosoftAI, @GoogleDeepMind, @Google)

Joined December 2007
Adam Sadovsky retweeted
Awesome seeing @JensenHuang talking about @harvey and @trajectorylabs work with Nemotron at GTC
7
34
4
127
11,254
Today we are announcing Harvey LAB-AA v1.1 in collaboration with Harvey. This updates our scoring methodology for the Legal Agent Benchmark (LAB) to add a hallucination check and require correct responses to not include material misstatements. LAB-AA v1.1's new headline metric, Hallucination-Gated All-Pass Rate, only credits a task when the deliverables satisfy every rubric criterion and contain no material hallucinations. We define a material hallucination as one that would mislead a reader on a substantive point, such as a wrong contractually required date, while a minor hallucination is a real error that is unlikely to meaningfully affect the legal interpretation of a deliverable. Minor hallucinations are reported separately and do not affect the headline score. Grok 4.7 (xhigh) leads at 9.4% Hallucination-Gated All-Pass Rate, and >60% of otherwise passing results across the models tested at launch contain a material hallucination. This is the first step in enhancing the methodology for Harvey LAB-AA. In future updates, we’re working with Harvey to better account for the full set of factors lawyers value, including usability features like style and tone. Key takeaways: ➤ Top of the leaderboard: Grok 4.7 (xhigh) from @spacexai leads at 9.4% Hallucination-Gated All-Pass Rate, narrowly ahead of Muse Spark 1.3 (max) from @AIatMeta at 8.9% and GPT-6 Astra (max) from @OpenAI at 8.6% ➤ Hallucinations reshape the leaderboard: without the hallucination gate, Muse Spark 1.3 (max) would lead clearly with a 26.7% all-pass rate, but two thirds of those passes contain one or more material hallucinations, resulting in a Hallucination-Gated All-Pass Rate of 8.9%. GPT-6 Astra (max) retains almost all of its passes after the hallucination gate (8.9% to 8.6%) and moves from joint 10th to 3rd ➤ The GPT-6 model family is the most grounded: GPT-6 Astra (max) averages 0.03 material hallucinations per task (4 across all 120 tasks) and GPT-6 Sol (max) averages 0.07. When comparing six checker models on a 20-task subset, GPT-6 Astra had zero material hallucinations under every checker, including Claude Opus 5.5 (high). By comparison, Gemini 3.8 Flash (high) averages 13.96 material hallucinations per task. ➤ Top-scoring models aren’t the most expensive: Grok 4.7 (xhigh) takes the top spot on the leaderboard at a cost of ~$9.50 per task, under half the cost of Claude Fable 5.1 (max with fallback) at ~$21.70, the most expensive model. Muse Spark 1.3 (max) represents strong performance for its cost, landing in second for ~$4.20 per task Thank you to @nikogrupen and @ItsJulioPereyra from @harvey for their work on Harvey LAB and collaboration.
37
56
83
571
949,432
Adam Sadovsky retweeted
When I shared @harvey’s model strategy a few months ago, there were two parts: 1. Build our own model 2. Use that to help customers do the same We’ve done the first. Now we’re hiring for the second: Harvey’s Private Model Program. We’re seeing huge demand from law firms to own their own intelligence leveraging private data. This will enable them to become frontier firms that get smarter with every client matter. You’ll:
 - Partner with law firms to build these systems. - Build the team that delivers this at scale.
 - Work with our technical org to define the platform that powers this team. We’re looking for a technical PM or founder type to own Harvey’s Private Model Program. DM me if this sounds interesting.
40
36
13
420
79,040
Adam Sadovsky retweeted
Seven new models launching at Build: let’s go! Reasoning. Code. Image. Transcribe. Voice. Built from scratch on a clean data lineage, designed for efficiency, working seamlessly as a family of models Thread 🧵 #MSBuild
137
511
122
3,351
404,239
Adam Sadovsky retweeted
big congrats to the microsoft AI team on MAI-Thinking-1! this is the kind of thoughtful post-training the field needs more of - focused on what actually matters to users excited to see a new frontier model in the race 😎 microsoft.ai/news/introducin…
1
29
1,129
Meet our third @MicrosoftAI model: MAI-Image-1 #9 on LMArena, striking an impressive balance of generation speed and quality Excited to keep refining + climbing the leaderboard from here! We're just getting started. microsoft.ai/news/introducin…
34
75
30
500
147,746
Adam Sadovsky retweeted
This was an amazing week at ⁦@MicrosoftAI⁩ !! We released MAI 1 preview and a taste of MAI Voice. I’m super happy with this team - only about 100 people and already shipping in ⁦@lmarena_ai⁩ in less than a year. Strong support. More soon. Thanks for feedback!
13
9
4
164
50,286
Adam Sadovsky retweeted
Just shipped a few updates 1. Gemini 2.5 Pro to try for free on gemini.google.com in the model drop down. Advanced has higher limits. 2. Canvas with 2.5 Pro in Advanced. Our best coding model yet. We had so much fun building demos internally, can't wait to see what y'all come up with!
Gemini 2.5 Pro is taking off 🚀🚀🚀 The team is sprinting, TPUs are running hot, and we want to get our most intelligent model into more people’s hands asap. Which is why we decided to roll out Gemini 2.5 Pro (experimental) to all Gemini users, beginning today. Try it at no cost at gemini.google.com
12
16
381
55,981
Adam Sadovsky retweeted
WE HAVE A NEW BEST MODEL IN THE WORLD! GEMINI 2.5 IS #1 ON LIVEBENCH
101
153
16
1,390
165,534
Adam Sadovsky retweeted
BREAKING: Gemini 2.5 Pro is now #1 on the Arena leaderboard - the largest score jump ever (+40 pts vs Grok-3/GPT-4.5)! 🏆 Tested under codename "nebula"🌌, Gemini 2.5 Pro ranked #1🥇 across ALL categories and UNIQUELY #1 in Math, Creative Writing, Instruction Following, Longer Query, and Multi-Turn! Massive congrats to @GoogleDeepMind for this incredible Arena milestone! 🙌 More highlights in thread👇
Think you know Gemini? 🤔 Think again. Meet Gemini 2.5: our most intelligent model 💡 The first release is Pro Experimental, which is state-of-the-art across many benchmarks - meaning it can handle complex problems and give more accurate responses. Try it now → goo.gle/4c2HKjf
71
387
108
2,294
470,516
Adam Sadovsky retweeted
If you're fine-tuning LLMs, Gemma 3 is the new 👑 and it's not close. Gemma 3 trounces Qwen/Llama models at every size! - Gemma 3 4B beats 7B/8B competition - Gemma 3 27B matches 70B competiton Vision benchmarks coming soon!
19
53
6
483
36,733
We replaced GPT-4o with Gemini-2.0 Flash for Bot9, reducing our costs by about 20× with no visible loss in accuracy. This change was implemented on a highly complex support agent that makes 32 tool calls. I was seriously not expecting this. At the application layer, it also made us one of the top 10 apps built with Gemini worldwide — and the only one from India in the list. - Data source : Openrouter.
39
63
17
1,231
130,091
1/ After residency at Mass General Hospital, I reported to Atlanta to meet my fellow CDC Epidemic Intelligence Service Officers. I have never felt so intimidated by my peers The best and the brightest, they were star clinicians, had served in disaster zones; MD/PhDs and MSF.
405
4,807
924
17,971
2,615,406
Adam Sadovsky retweeted
Andrej Karpathy
@karpathy30 Jan 2025
We have to take the LLMs to school. When you open any textbook, you'll see three major types of information: 1. Background information / exposition. The meat of the textbook that explains concepts. As you attend over it, your brain is training on that data. This is equivalent to pretraining, where the model is reading the internet and accumulating background knowledge. 2. Worked problems with solutions. These are concrete examples of how an expert solves problems. They are demonstrations to be imitated. This is equivalent to supervised finetuning, where the model is finetuning on "ideal responses" for an Assistant, written by humans. 3. Practice problems. These are prompts to the student, usually without the solution, but always with the final answer. There are usually many, many of these at the end of each chapter. They are prompting the student to learn by trial & error - they have to try a bunch of stuff to get to the right answer. This is equivalent to reinforcement learning. We've subjected LLMs to a ton of 1 and 2, but 3 is a nascent, emerging frontier. When we're creating datasets for LLMs, it's no different from writing textbooks for them, with these 3 types of data. They have to read, and they have to practice.
378
1,732
203
11,747
723,051
As @swyx’s Pareto frontier graph shows, Gemini is arguably the real story of the past few months
25
72
34
572
463,820
Our latest update to our Gemini 2.0 Flash Thinking model (available here: goo.gle/4jsCqZC) scores 73.3% on AIME (math) & 74.2% on GPQA Diamond (science) benchmarks. Thanks for all your feedback, this represents super fast progress from our first release just this past Dec! Latest version also includes code execution, a 1M token content window & a reduced likelihood of thought-answer contradictions. We’ve been pioneering these types of planning systems for over a decade, starting with programs like AlphaGo, and it is exciting to see the powerful combination of these ideas with the most capable foundation models.
121
349
93
2,561
696,338
Adam Sadovsky retweeted
Nice paper from Deepmind takes a fresh angle on factuality: arxiv.org/abs/2501.03200 While most existing factuality datasets focus on public world knowledge, this paper evaluates whether responses are consistent with a provided document as context. This is an elegant and tractable problem formulation because the surface area is well-specified and there are no disputes about what the ground truth should be. This type of factuality is hugely important for personalizing LLMs, which is a direction that the field has not been focusing on so far. Current LLM products (for the most part) are used in a context-agnostic way, for example, answering facts about the world or debugging a small piece of code. But a huge piece of intelligence is the ability to understand and follow the context for a very specific use case. Staying true to context is required not just for tasks like software engineering but even for tasks that don’t require lots of specialized training, such as taking someone’s fast food order. A language model that is able to totally not hallucinate given a long piece of context seems like a goal that we should be able to get to soon that will be transformative for these types of use cases
18
107
8
706
66,126
Adam Sadovsky retweeted
With the new reasoning models, there is increasing value in coming up with good questions. That reminded me of an old short scifi story about a future where coming up with new PhD ideas in a world of AI was hard I couldn't remember the name. Only Gemini Deep Research found it.
17
34
8
501
71,704
Adam Sadovsky retweeted
The new Deep Research feature from Google feels like one of the most appropriately "Google-y" uses of AI to date, and is quite impressive. I've had access for a bit and it does very good initial reports on almost any topic. The paywalls around academic sources puts some limits.
38
138
15
1,148
171,391