Jeonghwan Kim @ NeurIPS 2026 (Atlanta) ✈️

Found a paper on end-to-end training of latent diffusion models. After discussing it with @XiaoyuW9401147, we both think that although training DiTs on ImageNet feels like a toy experiment nowadays, the end-to-end training in this paper is still worth a look because it’s simple and elegant. Since the release of REPA (arxiv.org/abs/2410.06940), the focus of the diffusion community has shifted from training large scale DiT to training DiT more efficiently. Motivated by the lack of generation ability of VAE space as it is designed for reconstruction, REPA-E (arxiv.org/abs/2504.10483) proposed end-to-end training the DiT and VAE for better input space that will speed up convergence. Its key finding is that diffusion loss cannot be backpropagated to VAE as it will cause latent space mode collapse. RAE (arxiv.org/abs/2510.11690) revolutionized the training paradigm by proving that one can directly train DiT on representation space like Dinov2 or SigLIP2. Though semantic space itself is beneficial for generation, it cannot remove the need for two stage training (vision encoder, generator). SpeeDiff proposed by this paper unlocks the secret of training end-to-end latent diffusion. All we need is a TPR loss, which computes a pixel-level reconstruction loss by decoding the predicted clean latent from an intermediate noisy state using Tweedie’s formula. With TPR loss, we can use the diffusion loss to update the VAE encoder from scratch, demolish the claims in REPA-E. With the clean formulation of end-to-end training, we can now let the generative model itself to decide what is the best input space for generation. Looking forward seeing whether this will work on large scale text-to-image experiment or video generation. But still, thinking from a broader picture, why does the community need end-to-end training of latent diffusion model? The existence of vision encoder means that the generation will always be lossy. Maybe the endgame is moving the noise before the encoder: diffusion happens in pixel space, the VAE becomes part of the denoiser, and end-to-end training decides how much to compress at each noise level. Surprisingly this is last year CVPR paper, and it was released on arxiv four days ago🤯 arxiv.org/abs/2610.36419v1