
In some ways I'm surprised there haven't been even more major results in math with the latest models. It seems like progress should happen first in verifiable domains, before progress in any domain that has to interact with the real world. But even there, progress isn't runaway.
Said another way: disproving conjectures is nice, but why don't we see 10x as many results? 100x? 1000x? When will that happen? Why isn't it happening now?
9
25
3,260
I've often thought about a world where training rights closely track human access: if a person can legally access content, a model should also be able to learn from it, and it should cost about the same. You can regulate infringing outputs in the normal way.
The funny thing about whether open-source models and knowledge-sharing slow progress is that we have a counterfactual in the decade prior to ~2020 where deep-learning and generative-modeling research was maximally transparent, and it led to the world today.
1
10
1,540
I can attest after using explicit sharding for a couple of months that I feel a deep sense of calm whenever I train models, knowing exactly where all my shards are ahead-of-time.
highly recommend you try out JAX's new Explicit Sharding API. its more intuitive in that for intermediate computation .sharding will print the actual sharding at that point so you don't have to add with_sharding_constraint everywhere, but its a bit more strict. you can mix-and-match via the new auto_axes transform.
I'm rejoining @GoogleDeepMind in NYC this week, looking forward to catching up with folks :)
I find myself manually ensembling LLMs pretty often e.g. compare approaches from r1, o1 pro and o3-mini-high, or go back and forth between Claude and Gemini, asking each for their take on feedback from a 'colleague'.
It would be nice if this was built directly into product.
7
1,772
More than anything, the R1 model and paper make me wonder how far along we'd be if everyone was still clamoring to shout their best ideas from the rooftops. I know it's naïve to think we could sustain open research of the kind we saw up to 2020 indefinitely, but still...
This is one of those idealized trends that you imagine in a perfect world, but then the results actually bear it out
First book of the year and it was a doozy. I think I have a soft spot for aerospace engineering--I liked this almost as much as 'Carrying the Fire'.
4
496
Trying out NNX having previously switched from Haiku to Linen.
Something something fool me once, something something I'll probably just get fooled again.
1
2
954
I like the Bayesian framing of reward-based post-training (i.e. reward-maximization with a KL penalty).
Up to an additive constant, reward functions are log-likelihoods, and the pre-trained model is a prior. Then the posterior target is the product of the likelihoods and prior (the prior KL-weighting can equivalently sharpen or smooth your likelihoods). Rewards can be hard for math or code verification, or soft for subjective preference.
This means post-training (of this kind at least) optimizes KL(model || posterior), whereas pre-training optimizes KL(data || model). It also means post-training is mode-seeking (as opposed to mode-covering like pre-training), so those rewards better be well calibrated.
(Figure from 'RL with KL penalties is better viewed as Bayesian inference', link below along with other useful references)








