David Grangier (@GrangierDavid) profile picture

ML research with practical impact.

Joined December 2019
Photo by David Grangier (@GrangierDavid)Photo by David Grangier (@GrangierDavid)Photo by David Grangier (@GrangierDavid)Photo by David Grangier (@GrangierDavid)Photo by David Grangier (@GrangierDavid)Photo by David Grangier (@GrangierDavid)Photo by David Grangier (@GrangierDavid)Photo by David Grangier (@GrangierDavid)Photo by David Grangier (@GrangierDavid)Photo by David Grangier (@GrangierDavid)
Something new! I joined @genesismolai to work on AI for drug discovery. After years in machine learning research, drug discovery felt like an impactful place to apply my expertise. It’s also a field with lots of exciting problems and plenty of room to explore ML methods.
11
1
2
60
3,779
#NeurIPS2025 Mixing different datasets to train your LLM? ✨ We can help you find the perfect blend! 📈 Few small-model experiments → scaling law fit → your optimal mixture. 🎯 Easy + efficient. Chat with us 💬 Poster #3414. Thu, Dec 4, 11am
We propose new scaling laws that predict the optimal data mixture, for pretraining LLMs, native multimodal models and large vision encoders ! Only running small-scale experiments is needed, and we can then extrapolate to large-scale ones. These laws allow 1/n 🧵
1
1
5
1,833
#ICLR #TrainLLMBetter Tomorrow, #soup of experts, an #hypernetwork conditioned on a simple description of the test distribution: adaptation without retraining (Modularity workshop Sunday). arxiv.org/abs/2502.01804 Still on today... CRISP Importance Sampling for LLM pretraining.
1
287
#ICLR #TrainBetterLM I am at ICLR, come to our posters for improved language model training! Recycle gradients for faster neural net training with AdEMAmix iclr.cc/virtual/2025/poster/… (Fri Apr 25, 10 am). 1/3
1
1
8
1,272
2/3 Importance sampling for better pretraining distribution with CRISP iclr.cc/virtual/2025/poster/… (Sat Apr 26, 10 am).
1
2
156
3/3 Mixture of experts on high latency networks with No Need to Talk iclr.cc/virtual/2025/poster/… (Thu Apr 24 3pm). Joint work with @MatPagliardini, @NasFilippova,@PierreAblin @olivia61368522, Skyler Seto, @angeloskath, Ronan Collobert
2
144
New paper! arxiv.org/abs/2410.03735 Clustered importance sampling to build specialist Language Models (LMs) 🤔 Build a specialist LM with very little specialist data 💡How? Generalist data + efficient, scalable importance sampling w/ @Olivia61368522+SkylerSeto+@PierreAblin 1/4
1
14
1
25
8,159
🚀Easy with clustered importance sampling: 1️⃣ cluster the generalist dataset, 2️⃣ resample the clusters w/ their prior from tiny specialist data, 3️⃣ Done! 🏁 3/4
1
1
242
⦿ Efficient, scalable approach on LM and Q&A domains. ⦿ Single & multitask. ⦿ Pretraining & continued pretraining. ⦿ Ablations on data size, model size... arxiv.org/abs/2410.03735 4/4
194
Ademamix optimizer for jax/pytorch: change one line of code, train your model faster.
🎇Official pytorch/jax implementation of Ademamix🎇 github.com/apple/ml-ademamix Drop-in replacement for AdamW, much faster LLM pre-training! 🚀🚀🚀🚀
4
10
1,082
Faster, better model training by reusing old gradients (>10k steps ago) with negligible extra computation? Count me in. arxiv.org/abs/2409.03137
Stop discarding your old gradients! Introducing AdEMAMix, a novel (first-order) optimizer capable of outperforming Adam. Let’s have a thread on momentum and the surprising relevance of very old gradients. A joint work with @GrangierDavid and @PierreAblin #ml #optimization 1/🧵
1
9
1
61
9,120
At ICML? Learn about our efficient projected language models! Adding capacity to a traditional language model improves accuracy but increases inference cost. How to avoid this? We propose a novel architecture, projected networks (PN).
3
5
26
2,904
2/2 PN is a high capacity network whose parameters can be linearly projected into a small network. This strategy enables both high capacity and efficient inference. See details at our poster on Friday morning and afternoon. openreview.net/forum?id=Wi88… icml-fm-wild.github.io
199
New language model work! In practice, LMs often face a double constraint (i) small inference budget + (ii) little application-specific data: (i) means small specialized models for inference; (ii) means using auxiliary generic data e.g. for pretraining 1/2 arxiv.org/abs/2402.01093
1
13
35
5,926
2/2 Findings: when the application-specific training budget is large, importance sampling is great. Otherwise, asymmetric models (big at train, small at inference e.g. mixture of experts or hyper-networks) are attractive, better than the popular distillation strategy.
1
1
259
With Angelos Katharopoulos, Pierre Ablin, Awni Hannun.
209
Efficient bilevel algorithm for training data selection arxiv.org/abs/2311.11973 #bilevel #data_selection #DomainAdaptation #distshift #llm #NeurIPS2023 Online algorithm for filtering large (pre)training sets with maximal impact on the targeted task. (1/3)
1
19
1
81
15,879
Large models are often trained on massive web datasets and a bit of target-task data. In this setup, it is 👍 to spend more train effort on specific parts of the large set. Our online algorithm maintains an auxiliary cheap filter model when training the large model. (2/3)
1
3
546
Our analysis proposes a simple test to check if our method applies to your problem. Chat with us at our poster at #neurips2023 DistShift workshop next week. Joint work with Pierre Ablin, Awni Hannun. (3/3)
3
425