Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion
CVPR 2024(2023)
Abstract
Sampling from the posterior distribution poses a major computational
challenge in solving inverse problems using latent diffusion models. Common
methods rely on Tweedie's first-order moments, which are known to induce a
quality-limiting bias. Existing second-order approximations are impractical due
to prohibitive computational costs, making standard reverse diffusion processes
intractable for posterior sampling. This paper introduces Second-order Tweedie
sampler from Surrogate Loss (STSL), a novel sampler that offers efficiency
comparable to first-order Tweedie with a tractable reverse process using
second-order approximation. Our theoretical results reveal that the
second-order approximation is lower bounded by our surrogate loss that only
requires $O(1)$ compute using the trace of the Hessian, and by the lower bound
we derive a new drift term to make the reverse process tractable. Our method
surpasses SoTA solvers PSLD and P2L, achieving 4X and 8X reduction in neural
function evaluations, respectively, while notably enhancing sampling quality on
FFHQ, ImageNet, and COCO benchmarks. In addition, we show STSL extends to
text-guided image editing and addresses residual distortions present from
corrupted images in leading text-guided image editing methods. To our best
knowledge, this is the first work to offer an efficient second-order
approximation in solving inverse problems using latent diffusion and editing
real-world images with corruptions.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined