Ass. prof. of Generative AI. PI of Generative Memory Lab (@DondersInst). Generative diffusion, language diffusion and statistical physics. AI realist.

Nijmegen, Nederland
Joined July 2011
1/?) As promised to Sander Dieleman (@sedielem), we’re finally excited to share: Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion We show that continuous diffusion can achieve very strong language modeling performance when operating directly on bitstreams, outperforming masked and uniform diffusion baselines, and essentially matching autoregressive models under our evaluation settings.
6
37
5
238
34,681
Luca Ambrogioni retweeted
Can a neural network learn ~2^100 LDPC (Low-Density Parity-Check) codewords from just 10-20 examples? 👉 Grokking in LLMs and memorization-to-generalization transition in diffusion models are related phenomena — the model learns true process of data generation from a small number of training samples. 👉 It is hard to study this phenomenon analytically on natural datasets. 👉 LDPC codeword generation is a precise mathematical process in which the codewords are constructed as solutions to parity check equations. 👉 We train a Dense Associative Memory with a tiny fraction of codewords from the full vocabulary, without giving the network the generation algorithm itself. The model manages to learn the entire vocabulary on its own. It turns out, you only need log(n) samples to learn the entire space of 2^n codewords. 👉 The network only needs approximately P=10-20 training examples to correctly learn the full vocabulary containing ~2^100 codewords. Our new paper with Andy Lucas: arxiv.org/abs/2610.27384. Let us know what you think!
3
9
2
57
2,422
Luca Ambrogioni retweeted
Hot take: the community keeps asking whether diffusion LLMs can beat AR. That's the wrong question. Diffusion is a sampling method, and it can sit on top of an AR model to make it faster instead of replacing it. Speculative decoding already speeds up AR, but it has two long-standing problems. Training a drafter takes a long time, and you have to build a complete pretraining corpus for it. Even then, token acceptance drops sharply on out-of-domain data. Uno @ssahoo_ offers an elegant way out: it decouples sampling training from capability training. There's no separate drafter. The model itself, plus one LoRA, drafts several tokens in a single forward pass, and the LoRA learns to match the model's own next-token distribution, so there's no drafter corpus to build and no labels to collect. We applied this to the Looped Transformer Nanbeige4.2-3B: 1 hour of LoRA training gives a lossless 2× speedup on agentic task. We haven't optimized anything yet, and we believe the acceleration can go much further. The takeaway: diffusion doesn't need to beat AR. Any model that can score its next token can get a fast diffusion mode in an hour. Page: fengsxy.github.io/LoopUno Lora Adapters: huggingface.co/liddlefish/Lo…
4
23
8
134
11,935
Luca Ambrogioni retweeted
For the Navier-Stokes proof: "On a technical level, the paper is unbelievable, maybe beyond what a human could have come up with but not beyond human understanding"
Since I'm now blocked by bluey from replying to this thread, I answer here by quote tweet the openAI construction has more to do with convex integration than anything else. In convex integration, the oscillations are typically built by hand and the iteration procedure creates wild weak solutions. The non-uniqueness construction of Albritton–Brué–Colombo used an unstable vortex to split solutions apart. The Córdoba–Martínez-Zoroa singularities amplify vorticity across scales by strain. In the openAI paper, a fluid instability is clever made so that the flow itself generates the oscillations: they grow by drawing energy from the background shear. The Reynolds stress then becomes the force that creates the collapsing smooth vortex. Using a physical instability to create the stress in this way has no precedent, and it is what produces a smooth force rather than a merely rough one. Some on these ideas are in the literature (e.g. Leibovich-Stewartson 1983) but the way they are combined is definitely insane and definitely nowhere in the literature previously. I also don't know of anyone suggesting that convex integration would play a role in the creation of a blow-up solution like this. On a technical level, the paper is unbelievable, maybe beyond what a human could have come up with but not beyond human understanding: there are several different groups of pde experts working on understanding it who say that they more or less understand now. I expect digestions soon td;lr: anyone saying that this OAI result is unsurprising is bullshitting sorry!
9
17
315
22,091
Luca Ambrogioni retweeted
PhD applications at @GatsbyUCL are open! Full funding available regardless of nationality. Includes ✨new✨ Jump Trading studentships in AI with potential application to finance, eligible for employment opportunities with Jump Trading during PhD. Deadline: 🚨November 9th🚨
Interested in a #PhD in machine learning & theoretical neuroscience? Applications for 2027 entry to the Gatsby Unit PhD programme now open! 💰 Fully-funded 4-year programme 🤝 Close links to Sainsbury Wellcome Centre & the ELLIS Unit @ UCL (@uclcsml) ℹ️ ucl.ac.uk/life-sciences/gats…
1
17
149
6,646
Luca Ambrogioni retweeted
1/5 Continuous diffusion may be better at discrete reasoning than standard sampling suggests. We took SRM's MNIST-Sudoku Hard checkpoint (arXiv 2502.21075) and changed ONLY the sampler: ❗ 0.8% → 65.7% valid boards solved, without retraining. Why does standard sampling fail?🧵
7
29
2
249
11,304
Luca Ambrogioni retweeted
The magic of flow map language models is that they capture dependencies between tokens in just 1-2 model passes. Using this as a drafter for speculative decoding makes a ton of sense, and it's exciting to see this working as well as it does! (Love the shuffle test as well btw!)
Announcing FlashMap: a new speculative decoding algorithm based on flow maps! >6x faster inference on Qwen3-8B 🧵 Project page: jshen.net/flashmap arXiv: arxiv.org/abs/2610.24365 w/ @Jay41969851 @_ryantlee @jonathanmay @cosmo_shirley
4
30
237
11,500
Luca Ambrogioni retweeted
A key difference between continuous and discrete diffusion for language is that the former is (in principle) able to represent uncertainty in intermediate noisy states (i.e. superpositions of possible outcomes). Binary discrete diffusion is another way to achieve this. Neat!
🚨 New paper! What if LLMs could reason like we do when solving Sudoku: keep a few candidates per cell and iteratively narrow them down? Introducing Binary Discrete Diffusion for language and reasoning tasks: it denoises sets of candidate tokens using binary representations! 🧵
6
26
2
200
11,756
Luca Ambrogioni retweeted
I like the intuition here: remove strong local correlations to focus on weaker longer-range correlations, which results in better representations to align to. That same intuition inspired masked-self prediction (arxiv.org/abs/1903.04933) for hierarchical generative modelling.
What makes a good representation for diffusion models? Surprisingly, more information isn't always better. In our new work, Purify Before You Align, we show that removing locally predictable information can substantially improve diffusion alignment. 🎥 A quick overview 👇 🧵 1/n
3
15
1
173
10,192
Luca Ambrogioni retweeted
🎊 Excited to give a tutorial on diffusion LLMs at @NeurIPSConf 2026 with @StefanoErmon @JCJesseLai @mittu1204 📆 Mon, Dec 7, 9 AM AEDT 📍 TikTok Ent Centre (Sydney, Australia) Topics covered: > A beginner friendly introduction to diffusion-LLMs > Designing frontier diffusion-LLMs > Using Continuous diffusion for LLMs
2
26
2
91
7,296
Luca Ambrogioni retweeted
Can flow language models scale to billions of parameters and handle real generative tasks? Introducing 🌊 Lafa (Large flow language model), a flow LM scaled up to 8B that matches AR quality, with 🚀 2.5× AR throughput at 7B. Co-led with @ArnavChavan6 @UdbhavBamba. Joint work with @wognsfjq96 @hongseu33 @AdtRaghunathan @jw9730
9
51
9
217
22,403
Luca Ambrogioni retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
134
781
143
6,395
763,038
Luca Ambrogioni retweeted
We’re hiring a summer research intern at the MIT–IBM Computing Research Lab in Boston. We are looking for a PhD student near graduation. We’re interested in non-autoregressive architectures for language generation and reasoning, including diffusion models, flows, and EBMs.
3
11
1
212
10,217
Luca Ambrogioni retweeted
🎉 Excited to bring our book The Principles of Diffusion Models to @NeurIPSConf 2026 as a tutorial—this time, all about diffusion LLMs! 🚀 🔹 How do ideas from continuous diffusion extend to language generation? 🔹 What are the latest advances and open frontiers in diffusion LLMs? From foundational principles to cutting-edge research. Stay tuned! 📆 Mon, Dec 7, 9 AM AEDT 📍 TikTok Ent Centre (Sydney, Australia) w/ organizers @ssahoo_ @mittu1204 @StefanoErmon
🎊 Excited to give a tutorial on diffusion LLMs at @NeurIPSConf 2026 with @StefanoErmon @JCJesseLai @mittu1204 📆 Mon, Dec 7, 9 AM AEDT 📍 TikTok Ent Centre (Sydney, Australia) Topics covered: > A beginner friendly introduction to diffusion-LLMs > Designing frontier diffusion-LLMs > Using Continuous diffusion for LLMs
2
15
1
73
3,375
Luca Ambrogioni retweeted
Can a language model be based purely on self-conditioning? 🤔 Yes! Meet LAW 👋 (a.k.a. Loopholing All the Way!) 🧵1/8
10
75
12
380
36,207
Luca Ambrogioni retweeted
🚀 Are you a PhD student pushing the boundaries of AI, robotics, graphics, or accelerated computing? NVIDIA's Graduate Fellowship offers up to $60K in funding, mentorship from NVIDIA researchers, and a summer research internship! ⏳ Apply by October 30! blogs.nvidia.com/blog/applic…
6
23
130
14,705
Likewise, some glimpses sometimes
confession: i've yet to have that moment of feeling that frontier LLMs are geniuses, or super-human at ~anything broadly coherent that i deeply prize
4
662
Luca Ambrogioni retweeted
THIS IS DISINFORMATION The PHP exploit Nate is talking about was found by AI-enabled defensive operations. There is no evidence of large scale exploitation. This is a huge win for defense running ahead of offensive.
It's striking that this vulnerability was found by Sol and Argon rather than being found and fixed in the wake of Mythos 6mo ago. Seems like a heavy strike against the "defense will run ahead of offense" theory, especially give that domains like bio are much harsher than cyber.
13
39
4
362
14,194
Luca Ambrogioni retweeted
Here we explore the design space of losses for Strong Stochastic Flow Maps improving FD Dino by 10-50% over the original implementation by using a random ODE approach.
Excited to announce the follow-up to our NeurIPS 2026 Oral: Improved Training of Strong Stochastic Flow Maps! We improve the training of SSFMs by unlocking derivative-based losses! Joint work with @antoniofrancaib @rmincam @ChristianKoke @Niklas_TR and @AlexanderTong7
6
46
2,727
Luca Ambrogioni retweeted
From T. Tao slides:
[Public Lecture] Math 2.0 Terence Tao (October 9, 2026) Slides: teorth.github.io/tao-web/sli… Recording: aimath.org/videos/terry-tao-…
5
3
3
50
57,218