Information-Theoretic Diffusion

Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation. Whether interpreted through the lens of variational models or differential equations, diffusion models require many steps of expensive computation to give accurate density estimates. We introduce a new mathematical foundation for diffusion models inspired by classic results in information theory that connects Information with Minimum Mean Square Error estimators, the so-called I-MMSE relations. We generalize the I-MMSE relations to \emph{exactly} relate the data distribution and optimal denoising, leading to an elegant refinement of existing diffusion bounds. This new insight improves density estimation for diffusion models and enables simultaneous modeling of both continuous and discrete probabilities with no additional cost.