Exploiting Diffusion Prior for Generalizable Pixel-Level Semantic Prediction
CVPR 2024(2023)
摘要
Contents generated by recent advanced Text-to-Image (T2I) diffusion models
are sometimes too imaginative for existing off-the-shelf property semantic
predictors to estimate due to the immitigable domain gap. We introduce DMP, a
pipeline utilizing pre-trained T2I models as a prior for pixel-level semantic
prediction tasks. To address the misalignment between deterministic prediction
tasks and stochastic T2I models, we reformulate the diffusion process through a
sequence of interpolations, establishing a deterministic mapping between input
RGB images and output prediction distributions. To preserve generalizability,
we use low-rank adaptation to fine-tune pre-trained models. Extensive
experiments across five tasks, including 3D property estimation, semantic
segmentation, and intrinsic image decomposition, showcase the efficacy of the
proposed method. Despite limited-domain training data, the approach yields
faithful estimations for arbitrary images, surpassing existing state-of-the-art
algorithms.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要