Super excited to share that FinePhrase has been selected as an Oral Spotlight at #COLM2026! I will present on Wednesday afternoon next week. First in Oral Session 4 at 3:30PM PT and then in Poster Session 4 between 4:30 and 6:30PM (Poster F-4-126). If you're in SF, I am looking forward to seeing you at the talk or in the poster session! What is FinePhrase all about? Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop FinePhrase, a 486-billion-token open dataset of rephrased web text. We show that FinePhrase outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community. Listen to the paper to learn more: alphaxiv.org/abs/2604.13977v… Thanks to all my collaborators without whom this work would not have been possible! @_gucciiiii, Michal Štefánik, @gui_penedo, @HKydlicek, @eliebakouch, @_lewtun, @edwardbeeching, @thibaudfrere, Colin Raffel, @lvwerra, @Thom_Wolf

Prior work studied pre-pretraining (PPT) on small models and token budgets only. We scale it up to 7B parameters and 100B pretraining tokens across 4 data mixtures and 5 PPT tasks. We conclude that PPT is beneficial at larger scale, saving at least 21B PT tokens at the 3B scale. Listen to our paper to learn more: alphaxiv.org/pdf/2609.39827