Making the hills LLMs can climb towards becoming Math Copilots. AIMO Prize Manager. https://xitter.cf/t.co/ir85qxw65J (Opinions my own.)
Joined January 2023
- Tweets235
- Following60
- Followers289
- Likes65
I foresee that claiming a new conjecture (an activity that is already well established in math, but not prominent) will gain in prominence in the near future when AI will slowly (limited by token cost) clean up most conjectures.
1/ Just returned from Aussois, France, where on Sept 3rd I gave a talk about what is likely the first really full open LLM that can also do IMO-level problems well (Bronze medal @ IM026). And by open, I mean REALLY OPEN: checkpoints, harness, the lot.
aitp-conference.org/2026/
2/ What is nice about our model is that it arose fully from the community, and we worked really hard to make sure that weights were uploaded pre-AIMO, a former IMO medallists checked solutions, and that we worked to the highest level of rigor.
More info at fieldsmodel.org
If we take a step back from the entire controversy surrounding Navier-Stokes, and OpenAI, it becomes clear that things like these will happen with commercial LLMs. The only way one can be sure is to use a local LLM, as your data stays with you!
The most open one trained on math is Pochi-32B, released through my Fields Model Initiative -> training mixes, checkpoints, etc. everything is there. (The idea of the Fields Model Initiative is to host community-based fully open-source LLMs that approach the level of mathematics a Fields medallist could do; it seems commercial LLMs already are close to this level.)
This level of opennesnofnour model vastly exceeds what open-weight models share and it makes it easy to adapt our model for other use cases (we just focused on IMO where we got a medal in what is I think the most rigorous eval to date).
The problem: This was a volunteer-led effort where we just had some compute provided to train the model. While there is a team there that can do this type of work, it is hard to scale this to research-level math on a volunteer basis, where a bunch of researchers take away some spare hours for this type of work. It is surprising we got this far.
I would like to clarify a few things:
1) The screenshot is my reaching out to Levent to coordinate our releases. I hope it’s clear from the message that we came in with the best possible intentions.
2) I never ever asked for Levent to be removed from authorship of his own work (as indicated by my text). I was surprised to learn during the call with Tristan that they had only solved Euler and not Navier-Stokes; after learning this we brainstormed possible paths forward. One option we discussed was that Tristan could be the lead author on a rewrite of OpenAI’s Navier-Stokes proof. It is in that context that I said “it would be simpler if Levent was not an Anthropic employee” because I felt it would be inappropriate for an Anthropic employee to author OpenAI’s work. Importantly it was admitted that internal Anthropic models had been used in their proof of Euler blowup; I therefore felt I could not consider Levent to be an independent academic. Another option I wanted to propose (but got cut short) is to offer access to our internal model so that they could try to finish their proof and bridge the gap between Euler and NS. Again I did not know how to navigate giving access to internal OpenAI IP to an Anthropic employee.
3) To reiterate it plainly: as my text clearly indicates, and as I said during our call, OpenAI's intention was to do everything possible to celebrate their mathematical achievements and the heroic efforts that they made on Euler. In the call I was immediately met with a litany of slander, including direct threats that if we were to announce Navier-Stokes he would immediately go to the press with a barrage of unfounded accusations. I refuted all these accusations but he replied “there is nothing you can do, I simply do not trust you”. I was confused why one would turn an incredible source for celebration (of their achievements!) into such bickering, which is when I said that I did not understand why one would risk their career [over unfounded accusations]. Genuinely, at that moment, I was trying to care for him and do a last ditch attempt to get a chance to give them all the credits that they deserve. I deeply apologize for this extremely poor choice of words, it is the opposite of what I was trying to convey. (I should say that I retracted them on the spot by the way.)
4) Overall, on a personal level, it was incredibly difficult to have these conversations. Levent refused to attend any of the meetings despite my repeated asking. As Sholto Douglas said, there will need to be coordination between Anthropic and OpenAI in the future; I felt I was doing a proxy negotiation with Anthropic while the Anthropic employee refused to directly participate.
It's that time of the month. A new LLM has been launched (GPT 5.6) and everyone is jumping on their favorite conjectures and open problems to see if the LLM can help solve it.
If this is the future of mathematics, it's kind of bleak.
def RiemannHypothesis : Prop :=
∀ (s : ℂ), riemannZeta s = 0 →
(¬ ∃ (n : ℕ), s = -2 * (↑n + 1)) →
s ≠ 1 →
s.re = 1 / 2
:)
Apparently ChatGPT needs 120 minutes for this question. :D
chatgpt.com/share/69a5f141-2…
Papers like these are important for people competing in big reasoning competitions like AIMO or ARC-AGI.
The problem is that if one takes a closer look, there are some issues with the impressive claims:
- MATH is an outdated benchmark by now
- the numbers don't add up. The last sentence on page 1 states "Qwen-2.5-7B-Instruct improves from 76% to 95% while training just 10,000 parameters". This conflicts with table 2, which in turn is also unclear, as the parameter count doesn't seem to match with the # column.
Simon Frieder retweeted
Asking an AI system for an opinion is never a good idea. I am withholding judgement in whether to be impressed - it really depends on a lot of details: how many mathematicians tried and failed to prove the problem before (impossible to quantify, but that would be a measure of difficulty), what techniques the proof that was found uses (is it merely an obvious application of a know theorem that mathematicians overlooked, or did it introduce a new solution technique), etc.
An expert in the field needs to answer this -- not me, and definitely not Grok. LLMs still don't know what they don't know.
This is way over the head of Grok, and it's "arguments" are very weak since they would apply to any other piece of autoformalized result.
Is mathematics a game that is still worth playing in the long term?
The Twittersphere abounds with examples of what LLMs can do in math -- optimism is sky-high.
(I don't quite share that optimism since (open-source) LLMs do not even manage to solve the all "simple" unseen problems we have over at the AI Math Olympiad with the LB being stuck at 44/50.)
If that optimism pans out, even more maths will be created (rather than read) in the near future. While at first it will be exciting to watch conjectures fall, I am wondering what personal motivation will be left in such a full-automation scenario to get good and study mathematics.
In a year, we will be living in a world of mathematical ... AI slop!
I'd hope things would be different, but where they are headed now, I fear many technical, and completely uninteresting results will flood the space. arXiv already had to stop accepting position papers, and the same will happen for technical, niche research papers rather soon. There will be an occasional jewel where AI genuinely helped (although just autoformalizing right now is not that exciting to the mainstream AI-sceptical mathematician), but most pebbles that fall out of LLMs won't be these jewels, they'll be cobblestones.
Why Lean automation alone will not automate math -- here's more to the "automation" story than what Lean can express.
1/ Having a strong Lean engine is definitely a nice thing to have -- but there are limits to what is natural to do in Lean (or any formal systems).
I'm reminded here of the nice example that Patrick Massot mentioned in his talk about the concept of ... limits in calculus [1] :)
If you want to formalize all possible variants of each type of limit at a point of a function in a "naive" way, there are many definitions to state: of the limit at a point with the limit equaling a number, of the limit at a point with that point remove equaling infinity, etc. If you count them, it seems to amount to actually 256 (!) definitions (see the 45m30s mark of the video).
There are two ways out of this: 1) In practice, no one gets taught all 256 definitions but rather one teaches the "abstract generator" behind the definition. 2) One abstract all these away by using ultrafilters, and then just teaches limits in the setting of ultrafilters.
Both ways then also help to reduce the sprawling number of plumbing lemmas (like composition of two lemmas) that one otherwise would help to prove about limits (which are 4096 according to the video, which seems plausible).
Both solutions are possible execute in Lean, but are awkward to perform.
[1] youtube.com/watch?v=1iqlhJ1-…
2023: It's hard to devise an LLM that solves a math problem.
2025: It's hard to devise a math problem that stumps and LLM.
(...this in the context of competitive math questions, but we'll also get to research-level math soon)
Simon Frieder retweeted
AIMO3 is full of surprises: week 2 (out of 21) just concluded. After a race in the first week that had us both biting our nails to see how quickly the leaderboard is rising and cheering for the progress of open-weight LLMs, the leaderboard suddenly ground to halt.
My reason for this somewhat provocative claim is that the nature of mathematics will transform by quite a bit once LLMs go mainstream for mathematicians. I see some posts here and there on X, but these are early adopters, mathematicians that are interested in ML. The large majority of research mathematicians hasn't yet much explored working with LLMs from what I gather speaking with professors at various universities.
Once LLMs go mainstream, I predict a lot of conjectures will be solved. This will generate a few years of excitement. But what comes next?
Probably a bit of fun will go out if mathematicians become curators of math knowledge (produced by LLMs) rather than producers of knowledge.
Some domains are already so well developed (parts of algebraic geometry), and the road to mastering these domains so long, that it may just become infeasible to even learn, as a human, what the frontier is - this will take out another bit of fun.
That's why, for humans, my long-term take is rather pessimistic: we will produce a lot more mathematical knowledge, more the ever before, but it won't be as fun anymore and probably fewer people will want to do it, and who is to say that the usual "escape to abstraction" that was often the way to resist automation, will resist LLMs?
A 5-point update on how the latest release of DeepSeek-Math-V2 affects the AIMO.
In the meantime, working hard to get this 700GB monster to run on my 8x H100 to test it.
Can't wait for Unsloth and some of the other quantization wizards to release smaller versions!
