Conor Durkan (@conormdurkan) profile picture
New York
Joined November 2016
Photo by Conor Durkan (@conormdurkan)Photo by Conor Durkan (@conormdurkan)Photo by Conor Durkan (@conormdurkan)Photo by Conor Durkan (@conormdurkan)Photo by Conor Durkan (@conormdurkan)Photo by Conor Durkan (@conormdurkan)Photo by Conor Durkan (@conormdurkan)Photo by Conor Durkan (@conormdurkan)Photo by Conor Durkan (@conormdurkan)Photo by Conor Durkan (@conormdurkan)
In some ways I'm surprised there haven't been even more major results in math with the latest models. It seems like progress should happen first in verifiable domains, before progress in any domain that has to interact with the real world. But even there, progress isn't runaway. Said another way: disproving conjectures is nice, but why don't we see 10x as many results? 100x? 1000x? When will that happen? Why isn't it happening now?
9
25
3,260
I've often thought about a world where training rights closely track human access: if a person can legally access content, a model should also be able to learn from it, and it should cost about the same. You can regulate infringing outputs in the normal way.
a 🌶️ solution to the relative efficiency of chinese vs US labs
11
2
3
5,452
Voice-in and text-out is just the most wonderful way to interact with a computer.
1
3
341
And kudos to whoever is responsible for the transcription model running in ChatGPT is, it's so good.
2
196
The funny thing about whether open-source models and knowledge-sharing slow progress is that we have a counterfactual in the decade prior to ~2020 where deep-learning and generative-modeling research was maximally transparent, and it led to the world today.
1
10
1,540
I can attest after using explicit sharding for a couple of months that I feel a deep sense of calm whenever I train models, knowing exactly where all my shards are ahead-of-time.
highly recommend you try out JAX's new Explicit Sharding API. its more intuitive in that for intermediate computation .sharding will print the actual sharding at that point so you don't have to add with_sharding_constraint everywhere, but its a bit more strict. you can mix-and-match via the new auto_axes transform.
2
20
2,095
I'm rejoining @GoogleDeepMind in NYC this week, looking forward to catching up with folks :)
5
1
143
15,290
I find myself manually ensembling LLMs pretty often e.g. compare approaches from r1, o1 pro and o3-mini-high, or go back and forth between Claude and Gemini, asking each for their take on feedback from a 'colleague'. It would be nice if this was built directly into product.
7
1,772
More than anything, the R1 model and paper make me wonder how far along we'd be if everyone was still clamoring to shout their best ideas from the rooftops. I know it's naïve to think we could sustain open research of the kind we saw up to 2020 indefinitely, but still...
2
9
1,687
There's something so earnest and endearing about Operator trying to do things
1
875
Dial-up and usage limits from my childhood rushing back
1
758
I wrote up some thoughts having worked on generative music over the past year. I talk a bit about where tech is now, how it's being used, and where it might go. Link below!
1
2
16
1,440
This is one of those idealized trends that you imagine in a perfect world, but then the results actually bear it out
1
3
448
First book of the year and it was a doozy. I think I have a soft spot for aerospace engineering--I liked this almost as much as 'Carrying the Fire'.
4
496
Trying out NNX having previously switched from Haiku to Linen. Something something fool me once, something something I'll probably just get fooled again.
1
2
954
I like the Bayesian framing of reward-based post-training (i.e. reward-maximization with a KL penalty). Up to an additive constant, reward functions are log-likelihoods, and the pre-trained model is a prior. Then the posterior target is the product of the likelihoods and prior (the prior KL-weighting can equivalently sharpen or smooth your likelihoods). Rewards can be hard for math or code verification, or soft for subjective preference. This means post-training (of this kind at least) optimizes KL(model || posterior), whereas pre-training optimizes KL(data || model). It also means post-training is mode-seeking (as opposed to mode-covering like pre-training), so those rewards better be well calibrated. (Figure from 'RL with KL penalties is better viewed as Bayesian inference', link below along with other useful references)
12
81
8
615
72,191