Benchmarks are a public good. Theres should be a foundation for the purpose of measuring intelligence so the incentives for making benches and selling data do not misalign
interesting things are happening with the AI benchmark companies
let's say you made a new benchmark for AI. fantastic!
there's only two possibilities:
1. your benchmark sucks/doesn't represent model performance because there's an issue with the task/harness, in which case it should be ignored
2. the benchmark is actually really good, and you made a bunch of genuinely hard tasks pushing frontier model ability
in the case of 2. then congrats, you've also accidentally become an RL task company, because there's no difference (at all) between frontier benchmarks and frontier RL tasks.
you realize you can sell these tasks for a LOT of money, and start scaling. oh no, you're Mercor!
also now that you've sold similar tasks to some labs (probably non-exclusive contracts) so your benchmark is now saturated and meaningless, also model ability has improved fast enough that it probably gets saturated anyway even if you don't sell the tasks
so remember, when it comes to frontier model benchmarking, there is only dogshit and Mercors, and nothing in between!
ever wonder why Mercor/Handshake/etc publish benchmarks all the time? well only they can. because they're actually building tasks that push frontier models, and the best way to show that is to make benchmarks and release the results.
to be clear, this applies to model benchmarks only. if you're benchmarking sandboxes/web search/scraping/gpus/what have you this rule does not apply. but even then if you're good at that why not dip your toe in the infinite money pool?
(some exclusions apply, like preference data which doesn't require task creation, hence Arena)
Why do docs pages just link to other docs pages instead of opening the app to the right page?
Like you know what im looking just show me where to make the changes
mirendil mentioned :D
Thomas Kurian: 'In addition to our work with Anthropic, both directly and via Broadcom, and public references customers like Apple, who are scaling pretraining of their foundation models on the latest generations of TPUs, we're also working with new frontier model companies including Mirendil and Physical Intelligence.'
Arno retweeted
Thomas Kurian: 'In addition to our work with Anthropic, both directly and via Broadcom, and public references customers like Apple, who are scaling pretraining of their foundation models on the latest generations of TPUs, we're also working with new frontier model companies including Mirendil and Physical Intelligence.'
Arno retweeted
EBITDA = Earnings Before I Trick Da Auditor
Anthropic has filed an S-1 registration statement with the SEC related to its proposed IPO.
Read more: anthropic.com/news/s1-sec
Legal disclaimer: A registration statement relating to these securities has been filed with the SEC but has not yet become effective. These securities may not be sold, nor may offers to buy be accepted, prior to the time the registration statement becomes effective. This post does not constitute an offer to sell or the solicitation of an offer to buy these securities, nor shall there be any sale of these securities in any state or jurisdiction in which such offer, solicitation or sale would be unlawful prior to registration or qualification under the securities laws of any such state or jurisdiction.
My friends can attest, Ive been wanting to push boundaries of harnesses for a while.
When I was introduced to mirendil I was immediately intrigued that a small neolab was planning to compete in the big leagues, in the domain of agentic intelligence. It sounded insane, ambitious, and a great place to create the next generation of ai tooling.
Like how a noun needs a verb, no matter how powerful the model gets we will always need a harness. The space isnt quite solved but its close. Harnesses are all aligning to the same few architectures (more to come for how we built ours). Each generation of model has made it more clear what architectures are actually required.
Our goal was fundamental and simple. Give the model agency. Give the human freedom.
Im excited to introduce Lor, our new product built from first principles to be an always-on, multi client, multi environment delegator for coding and experiments.
We're opening up our beta. I'd love to chat with anyone pushing the limits of engineering and discovery.
Message me on any platform or sign up here if you're interested in trying it out
mirendil.com/lor-beta/
What would you do with 10M context?
Arno retweeted
"claude, take a serious stab at pretraining. here's $10m"
Today, we're sharing a major milestone in our mission to democratize frontier AI development:
Pretraining at scale without a pretraining team.
We set out to answer an ambitious question: Can we significantly reduce the time, human expertise, and compute needed to pretrain a competitive foundation model from scratch?
The result is Mirendil-Base, our first pretrained coding model, built by one researcher in just 3.5 months with less than $10M in compute, including all research, experimentation, and the final training run.
Our approach is already showing promising results as we scale to even larger models.
This is an important step toward a future where a single researcher, empowered by AI, can accomplish what once required an entire research organization. As these capabilities improve, we believe frontier AI development can become accessible to far more people and teams.
More to come soon, including independent evaluations, post-training benchmarks, and API access.
Arno retweeted
Mirendil pretrained its first strong coding model with a single researcher in 3.5 months and less than $10M in compute. This is an exciting step toward more accessible frontier AI, with much more to come.
We formally announced Mirendil less than 4 months ago. Today, I'm excited to share an important milestone for:
Pretraining at scale without a pretraining team.
Mirendil's mission is to solve Recursive Self-Improvement (RSI) and use it to democratize frontier AI development, ultimately accelerating science and technology. At Mirendil, when we think about RSI, we think holistically about the company as an intelligent entity that self-improves toward its goal. From that perspective, the most powerful notion of acceleration is reducing the time and resources (number of human experts, compute, and other costs) needed to achieve ambitious goals.
Given that pretraining is perhaps the most important yet most challenging part of frontier AI development, we decided to run an ambitious controlled experiment: Starting fully from scratch, can we significantly reduce the time, number of human experts, and compute resources needed to go all the way to successfully pretraining a strong coding model at scale?
Mirendil-Base is the result of this experiment. Our first pretrained model was built by one researcher in 3.5 months, with less than $10M in compute. This includes all the research and small-scale experiments it took to get there, as well as the final training run.
The recipe holds up as the models get bigger. For reference, Mirendil-Base is in the 100B-500B parameter range, Mirendil-Base-Large in the 500B-2T range, and Mirendil-Base-XL in the 2T-10T range.
We're beginning to see a world where a single researcher, empowered by AI, can take on work that once required an entire research organization. As these systems improve, the bottleneck shifts from the size of a team to the ambition and judgment of the people directing them. This has profound implications for who gets to build AI and, ultimately, who gets to push the frontiers of science and technology forward.
We’ll share independent evaluations, post-training benchmarks and API-access sooner than you think so stay tuned!
Arno retweeted
Thanks Samira. You are right. Just updated the plot with Beam and Muse that recently came out. Beam is a strong model!
Why do people that know how train reap the most benefits of ai.
Its good to recognize that we are in a transition period where the world is warming up to what AI can do. Right now the knowledge to train models is locked up with a small group of people.
We want anyone interested in scaling their expertise to be able to. AI was never the goal, just a means. It should be possible to create and spread knowledge for the benefit of all.
We formally announced Mirendil less than 4 months ago. Today, I'm excited to share an important milestone for:
Pretraining at scale without a pretraining team.
Mirendil's mission is to solve Recursive Self-Improvement (RSI) and use it to democratize frontier AI development, ultimately accelerating science and technology. At Mirendil, when we think about RSI, we think holistically about the company as an intelligent entity that self-improves toward its goal. From that perspective, the most powerful notion of acceleration is reducing the time and resources (number of human experts, compute, and other costs) needed to achieve ambitious goals.
Given that pretraining is perhaps the most important yet most challenging part of frontier AI development, we decided to run an ambitious controlled experiment: Starting fully from scratch, can we significantly reduce the time, number of human experts, and compute resources needed to go all the way to successfully pretraining a strong coding model at scale?
Mirendil-Base is the result of this experiment. Our first pretrained model was built by one researcher in 3.5 months, with less than $10M in compute. This includes all the research and small-scale experiments it took to get there, as well as the final training run.
The recipe holds up as the models get bigger. For reference, Mirendil-Base is in the 100B-500B parameter range, Mirendil-Base-Large in the 500B-2T range, and Mirendil-Base-XL in the 2T-10T range.
We're beginning to see a world where a single researcher, empowered by AI, can take on work that once required an entire research organization. As these systems improve, the bottleneck shifts from the size of a team to the ambition and judgment of the people directing them. This has profound implications for who gets to build AI and, ultimately, who gets to push the frontiers of science and technology forward.
We’ll share independent evaluations, post-training benchmarks and API-access sooner than you think so stay tuned!
Not enough people follow this guy.
Youve probably used some of his products like,
Gmail
Google docs
For example 👀
