🏔️
Joined October 2021
- Tweets95
- Following574
- Followers250
- Likes5.6K
Pinned Tweet
🎉 Excited to share our latest work has been accepted at #ICML2025.
We explain how Lottery Ticket masks can be extended to arbitrary random initializations by leveraging permutations of weight symmetries achieving strong generalization.
Check out the details: arxiv.org/abs/2505.05143
See you all in Vancouver!😃
1/10 🧵
🔍Can weight symmetry provide insights into sparse training and the Lottery Ticket Hypothesis?
🧐We dive deep into this question in our latest paper, "Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry", accepted at #ICML2025
“more capable than GPT-6 Astra” 👀
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
Rohan Jain retweeted
Checking that a major mathematical proof is correct can take years. Formalization—converting the mathematical reasoning into a form computer proof assistants like Lean can verify—can help.
Last month, Claude completed the first formalized proof of Fermat’s Last Theorem, one of the most famous theorems of all time. This was a project experts thought would take many years. It is the largest Lean proof ever written.
Fermat’s Last Theorem was first proven in 1995 by Sir Andrew Wiles, more than 350 years after it was conjectured. Our proof, which totals over 13 million lines of code, provides machine verification. More importantly, it proves over 29,000 other theorems that the proof requires, across many areas of math which had never before been formalized.
We see this as a major step in the long process of firming up the core of mathematical knowledge, building on work from three centuries of mathematicians and hundreds of contributors to Lean and Mathlib. We are optimistic that AI-assisted verification of mathematical proofs will help reduce the burden of refereeing mathematics in an era where more proofs are being produced than ever before.
You can read about the process on our Science Blog: anthropic.com/research/forma…
And see the complete proof on GitHub: github.com/anthropics/fermat…
Congrats Jacob! I still drop by the MAT475 problems every now and then for old time's sake. Always loved the enthusiasm and unique perspectives you brought to class.🤘
Link to the problems if anyone's interested: math.utoronto.ca/~jacobt/MAT…
‘Nobel Prize of mathematics’: @UofTMath's Jacob Tsimerman awarded prestigious Fields Medal #UofT 🏅 uoft.me/jacob
ALT Jacob Tsimerman receives the Fields Medal (photo by Allie Ippolito)
🎉 Come check out our work at #ICML2026!
Normalization layers can slow sparse neural network training by skewing gradients. We introduce SparseOpt to fix it and speed up convergence. Details in thread 👇
[1/5]🧵
🔍 Normalization layers have become ubiquitous in modern deep learning architectures, but do they always improve training?
In our #ICML2026 paper, we show that, in sparse neural network training, Normalization Layers can actually hurt optimization and slow down convergence.
If you're attending ICML, stop by our poster!
📍 Poster #4105
🗓️ Tue, Jul 7, 2026
🕑 2:00–3:45 PM KST
📍 Hall A
Rohan Jain retweeted
Looking for a fun weekend read?
Introducing the Illustrated ICML 🌎
We indexed all 6000+ ICML papers and built a visual way to explore the whole landscape
Search your favorite topics, open a cluster, and dive right in
Inspired by @JayAlammar
Need to sit down and digest this 🤯
AI has now solved a major open problem -- one of the best known Erdos problems called the unit distance problem, one of Erdos's favourite questions and one that many mathematicians had tried.
openai.com/index/model-dispr…
2
229
Rohan Jain retweeted
I expect optimization methods to be among the first areas of math to greatly benefit from AI. There are thousands of papers with optimizations methods and their convergence proofs that have a surprisingly standardized structure.
Most ideas fall somewhere in the range of momentum, averaging, adaptive learning rates, schedulers, weight decay, combined with choices of norms, usually one of {ℓ₂, ℓ∞, spectral}. I think that a combination of these is sufficient to derive the next sota in optimizers.
The hard part is getting something that works well in practice across different domains with universal hyperparameters. But the benchmarks are easy to set up, and with a constant feedback loop, agents are likely to discover a lot of new optimizers, one of which will turn out to work well. The potential savings in compute definitely make it worth trying.
Exciting read!
1/ Deep learning is going to have a scientific theory. We can see the pieces starting to come together, and it's looking a lot like physics!
We're releasing a paper pulling together these emerging threads and giving them a name: learning mechanics.
🔨 arxiv.org/pdf/2604.21691 🔧
3
306
Rohan Jain retweeted
We made Muon run up to 2x faster for free!
Introducing Gram Newton-Schulz: a mathematically equivalent but computationally faster Newton-Schulz algorithm for polar decomposition.
Gram Newton-Schulz rewrites Newton-Schulz such that instead of iterating on the expensive rectangular X matrix, we iterate on the small, square, symmetric XX^T Gram matrix to reduce FLOPs. This allows us to make more use of fast symmetric GEMM kernels on Hopper and Blackwell, halving the FLOPs of each of those GEMMs.
Gram Newton-Schulz is a drop-in replacement of Newton-Schulz for your Muon use case: we see validation perplexity preserved within 0.01, and share our (long!) journey stabilizing this algorithm and ensuring that training quality is preserved above all else.
This was a super fun project with @noahamsel, @berlinchen, and @tri_dao that spanned theory, numerical analysis, and ML systems! Blog and codebase linked below 🧵
Rohan Jain retweeted
Today's blog discusses how to project a neural network's (high-dim) loss landscape to 2D, while preserving local information, so that the "a ball rolling down a hill" picture is realistic and insightful -- we can clearly see why/when stick plateaus happen
kindxiaoming.github.io/blog/…
Rohan Jain retweeted
Does SGD really “seek flat minima”?
We show that SGD has no intrinsic preference for flatness, even for stable linear networks—going against ~10 years of folklore.
Flatness emerges iff label noise is isotropic; anisotropic noise drives SGD to arbitrarily sharp solutions.
This reveals a new flattening–sharpening mechanism in late training, unrelated to standard progressive sharpening or Edge-of-Stability effects.
Rohan Jain retweeted
Why don’t neural networks learn all at once, but instead progress from simple to complex solutions? And what does “simple” even mean across different neural network architectures?
Sharing our new paper @iclr_conf led by Yedi Zhang with Peter Latham
arxiv.org/abs/2512.20607
Congrats to my buddy @rajpalleti314 and his team! This is HUGE!
We just raised a $7M Seed round co-led by @MenloVentures and @haystackvc with participation from @Shakti_VC, @conviction and @upfrontvc 🚀
We're honored to have the support of incredible angels including @ericschmidt, @SebastianThrun, @sarahookr
Join us: alphaxiv.org/careers
6
712
Rohan Jain retweeted
📷📷📷New paper! (with @OpenAI) 📷📷📷
We trained weight-sparse models (transformers with almost all of their weights set to zero) on code: we found that their circuits become naturally interpretable! Our models seem to learn extremely simple, disentangled, internal mechanisms!
Rohan Jain retweeted
There’s lots of symmetry in neural networks!
🔍 We survey where they appear, how they shape loss landscapes and learning dynamics, and applications in optimization, weight space learning, and much more.
➡️ Symmetry in Neural Network Parameter Spaces arxiv.org/abs/2506.13018
Rohan Jain retweeted
🎯 Just released a new preprint that proves LR transfer under μP.
-> The Problem: When training large neural networks, one of the trickiest questions is: what learning rate should I use? [1/n]🧵
Link: arxiv.org/abs/2511.01734
Check it out! 👇
We now have a blog post up on explaining our ICML 2025 paper:
calgaryml.com/blog/2025/spar…
This blog is based on my recent talks in the UK and Germany, including at @microsoft, @imperialcollege, the CISPA Helmholtz Center for Information Security, and @UniofBath
4
612