Denoising Diffusion Probabilistic Models

The paper investigates diffusion probabilistic models, a class of latent variable models inspired by nonequilibrium thermodynamics, for high-quality image synthesis. The authors establish a novel connection between these models and denoising score matching with Langevin dynamics, which leads to a weighted variational bound for training. The models achieve state-of-the-art results on the CIFAR10 dataset with an Inception score of 9.46 and an FID score of 3.17, and comparable quality to ProgressiveGAN on the 256x256 LSUN dataset.

The diffusion model is defined as a parameterized Markov chain trained using variational inference to reverse a diffusion process that adds noise to data. The reverse process is learned to produce samples matching the data distribution. The authors propose a parameterization that reveals an equivalence with denoising score matching and annealed Langevin dynamics, which is key to achieving high sample quality.

Despite the high sample quality, the models do not achieve competitive log likelihoods compared to other likelihood-based models. The authors find that the majority of the model's lossless codelengths are consumed by imperceptible image details. They analyze this in terms of lossy compression and show that the sampling procedure resembles autoregressive decoding, suggesting a potential for progressive lossy compression.

The paper suggests that diffusion models have an inductive bias that makes them excellent for lossy compression, as evidenced by their high sample quality despite lower log likelihoods. The authors propose future work to explore the utility of diffusion models in other data modalities and as components in other generative models. They also highlight the potential for diffusion models in data compression and representation learning.

The authors acknowledge the broader impact of their work, noting that while diffusion models can improve generative modeling, they also pose risks such as the creation of fake images and reinforcement of dataset biases. They emphasize the importance of considering these implications as the technology advances.