Thejas Venkatesh retweeted
Introducing Samaya Sterling, our new model post-trained for challenging financial research that outperforms the frontier! Sterling started off as a two person research effort to see if we could increase efficiency in agentic orchestration. To our surprise we were able to post-train a much smaller model (Nemotron 3.5 Lightning, a 30B mixture-of-experts model with 3B active parameters) to greater accuracy (58.9% vs 55.9% on FrontierFinance) and 11x lower cost ($0.61 vs $6.89) than Claude Fable 5.1, the best performing frontier model.
We did this via a four-stage training process, two stages of SFT where the model first learns to research with our financial tools and how to construct a good agentic research strategy. The second two stages are RL, with an offline GRPO stage where the model further hones its research strategy from eight teachers that provide diverse approaches, and an online GRPO stage where the model can make end to end attempts and learn from its mistakes.
We see substantial gains from the co-adaptation of the model and harness through post-training, both through the model learning to effectively use the high-quality financial tools, and better compaction approaches that let even a small model execute on longer and deeper research.
We are exploring multiple follow-up directions, from multi-agent research swarms using this small model, to training firm-specific models to encode unique judgement, to training a larger model, where early results show even greater gains from the training process.
The full research is linked below!
Thejas Venkatesh retweeted
Samaya trained Nemotron 3.5 Lightning to beat Fable 5.1 on its finance research benchmark at 9% of the cost. The model was trained on Tinker together with the harness it works in: first by imitating 8 teachers, then RL on its own mistakes.
Introducing Samaya Sterling: our first post-trained model for state-of-the-art financial research.
On FrontierFinance, Sterling outperforms Claude Fable 5.1 (58.9% vs. 55.9%) at 11× lower cost.
Using a model with just 3B active parameters, Sterling demonstrates what's possible when specialized training meets a purpose-built financial research harness.
Read the full research ↓
Really excited to share @samaya_AI's first post-trained model, Sterling!
It’s been a great experience working on this. We’ve made some interesting discoveries along the way. Much more to come soon!
Joint work with @ozankoyluoglu, @richarddm1, @AshwinParan, and @yuhaozhangx.
Introducing Samaya Sterling: our first post-trained model for state-of-the-art financial research.
On FrontierFinance, Sterling outperforms Claude Fable 5.1 (58.9% vs. 55.9%) at 11× lower cost.
Using a model with just 3B active parameters, Sterling demonstrates what's possible when specialized training meets a purpose-built financial research harness.
Read the full research ↓
8
188
Drop by to chat about our work on deep research agents! :) @HowardYen1 @AshwinParan @yuhaozhangx @samaya_AI
Presenting the poster at COLM tomorrow! Please drop by if you want to chat :)
⏰ Tuesday 4:30 - 6:30pm
🍭 Imperial Ballroom (# 20)
3
65
Thejas Venkatesh retweeted
We partnered with @AnthropicAI to evaluate @claudeai Fable 5.1 pre-launch on FrontierFinance! Fable 5.1 becomes the best performing frontier model by a wide margin, scoring 55.9% compared to 49.2% for Fable 5 (the prior best performing frontier model.)
This gain largely comes from Fable 5.1 carrying out more and better tool calls, e.g. calling and anchoring better on authoritative sources.
The increased tool calls does come with increased cost though -- we observe Fable 5.1 is about 1.7x as expensive as Fable 5.
Exciting results and lots more work ahead to continue advancing agentic capabilities!
Thejas Venkatesh retweeted
A month ago we released FrontierFinance, an open benchmark for the frontier capabilities of finance AI agents. The response from community has been terrific, and the latest models are already making fast progress.
Today we're publishing the technical report: our annotation process, a deep analysis of the dataset, and a close look at how current systems and models actually behave, with insights we hope will shape the next generation of frontier models and harnesses.
Full report 👇
Thejas Venkatesh retweeted
Today we are releasing @samaya_AI's FrontierFinance benchmark. It is a fully open, hard benchmark that tests frontier intelligence of finance AI systems. We release 220 queries and a total of 11,543 rubrics, all crafted by finance experts.
Highlights in 🧵:
Thrilled to share FrontierFinance, our most ambitious benchmark yet for evaluating agentic performance in finance.
It’s been great to contribute to this release. Check it out!
Excited to be releasing FrontierFinance, the largest and most challenging open benchmark for evaluating AI agents across the full investment workflow!
FrontierFinance is substantially harder than current finance benchmarks: Existing benchmarks like FinanceBench and Finance Agent focus almost entirely on data extraction.
FrontierFinance spans diverse use cases across the full investment process: Screening & Discovery, Company Research, Sector/Industry/Macro, Earnings & Events, and Coverage & Catalyst Monitoring.
Created for ambiguous, long-horizon agents: 220 examples paired with 11,543 expert-crafted rubrics, following Samaya's Criteria Eval methodology. The rubrics are what let us evaluate the reasoning and steps behind a true expert-level output, not just a plausible-looking one.
Evaluations: We evaluated Claude Fable 5, Claude Opus 4.8, GPT 5.5, Gemini, open-source models including GLM and DeepSeek, and others. We used the same public rubric and a standard harness for financial tasks. Samaya's AI system reached state-of-the-art accuracy at 50.8%, at 4x lower inference cost than Fable 5. Next best was Fable 5 (49.2%), then Opus 4.8 (45%) and GPT 5.5 (43.5%).
We're releasing the benchmark, methodology, and full evaluation results - see link in comments.
Future releases: FrontierFinance was curated from Samaya's larger internal set of ~5,000 examples, and we plan to release subsequent, harder benchmarks as well as a more detailed technical report!
1
8
1,630
Insane throwback 😂 @pHequals7
can now proudly say i shared many a stage with ramp's newest CTO🔥
chennai bois always stay winning..
3
672
Thejas Venkatesh retweeted
@Samaya_AI trained a query understanding model that outperforms Claude Sonnet 4.6 and Gemini Flash on financial queries, with lower latency.
Better query understanding means better agents.
Here's how we trained it and the results 👇
@AshwinParan @_vThejas @ozankoyluoglu @yuhaozhangx @richarddm1
Had a fantastic time presenting Samaya AI's research at @CAISconf with @SkylerHallinan!
It was great collaborating on this project and sharing our work with the community.
Kudos to @lateinteraction, @heathercmiller and @aviaviavi__ for organizing a fantastic inaugural ACM CAIS!
Congratulations to the entire team at Samaya AI and USC who made this work possible: @AshwinParan, @yuhaozhangx, @jmhessel, @xiangrenNLP, and @Saipraneeth_R.
1
3
149
Thejas Venkatesh retweeted
.@SkylerHallinan presenting our work on OpaqueToolsBench at CAIS today. We look at how to measure and improve tool documentation for tools that are inherently hard or even impossible to describe without rollouts. The hardest bit was creating the benchmark!
Thejas Venkatesh retweeted
How to build agentic search systems for long-horizon tasks?
Check out our new paper!
- Simple design principles are efficient and effective
- Error analysis and fine-grain analysis for search systems
A 🧵 on SLIM, our long-horizon agentic search framework
Thejas Venkatesh retweeted
🚀Excited to share that our paper "ColBERT-serve: Efficient Multi-Stage Memory-Mapped Scoring" was accepted to #ECIR2025. Huge thanks to my amazing collaborators! Looking forward to insightful discussions—let’s connect! 🇮🇹🔍 #NLP #InformationRetrieval
Thejas Venkatesh retweeted
Handling uncertainty and messy real-world information is where LLMs fall short. That's why at Samaya AI, we are pioneering Causal World Models—AI systems that don't just predict but understand cause and effect in complex domains.
This approach enables AI to reason through uncertainty, generate better hypotheses, and work alongside experts to unlock deeper insights—whether in financial research, strategic decision-making, or beyond.
Read more about how we are pushing the boundaries of AI reasoning ⤵️























