Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering
CoRR(2024)
Abstract
Generating realistic images from arbitrary views based on a single source
image remains a significant challenge in computer vision, with broad
applications ranging from e-commerce to immersive virtual experiences. Recent
advancements in diffusion models, particularly the Zero-1-to-3 model, have been
widely adopted for generating plausible views, videos, and 3D models. However,
these models still struggle with inconsistencies and implausibility in new
views generation, especially for challenging changes in viewpoint. In this
work, we propose Zero-to-Hero, a novel test-time approach that enhances view
synthesis by manipulating attention maps during the denoising process of
Zero-1-to-3. By drawing an analogy between the denoising process and stochastic
gradient descent (SGD), we implement a filtering mechanism that aggregates
attention maps, enhancing generation reliability and authenticity. This process
improves geometric consistency without requiring retraining or significant
computational resources. Additionally, we modify the self-attention mechanism
to integrate information from the source view, reducing shape distortions.
These processes are further supported by a specialized sampling schedule.
Experimental results demonstrate substantial improvements in fidelity and
consistency, validated on a diverse set of out-of-distribution objects.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined