In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing
CVPR 2024(2024)
Abstract
3D-aware GANs offer new capabilities for view synthesis while preserving the
editing functionalities of their 2D counterparts. GAN inversion is a crucial
step that seeks the latent code to reconstruct input images or videos,
subsequently enabling diverse editing tasks through manipulation of this latent
code. However, a model pre-trained on a particular dataset (e.g., FFHQ) often
has difficulty reconstructing images with out-of-distribution (OOD) objects
such as faces with heavy make-up or occluding objects. We address this issue by
explicitly modeling OOD objects from the input in 3D-aware GANs. Our core idea
is to represent the image using two individual neural radiance fields: one for
the in-distribution content and the other for the out-of-distribution object.
The final reconstruction is achieved by optimizing the composition of these two
radiance fields with carefully designed regularization. We demonstrate that our
explicit decomposition alleviates the inherent trade-off between reconstruction
fidelity and editability. We evaluate reconstruction accuracy and editability
of our method on challenging real face images and videos and showcase favorable
results against other baselines.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined