Geometry-aware Reconstruction and Fusion-refined Rendering for Generalizable Neural Radiance Fields
arxiv(2024)
摘要
Generalizable NeRF aims to synthesize novel views for unseen scenes. Common
practices involve constructing variance-based cost volumes for geometry
reconstruction and encoding 3D descriptors for decoding novel views. However,
existing methods show limited generalization ability in challenging conditions
due to inaccurate geometry, sub-optimal descriptors, and decoding strategies.
We address these issues point by point. First, we find the variance-based cost
volume exhibits failure patterns as the features of pixels corresponding to the
same point can be inconsistent across different views due to occlusions or
reflections. We introduce an Adaptive Cost Aggregation (ACA) approach to
amplify the contribution of consistent pixel pairs and suppress inconsistent
ones. Unlike previous methods that solely fuse 2D features into descriptors,
our approach introduces a Spatial-View Aggregator (SVA) to incorporate 3D
context into descriptors through spatial and inter-view interaction. When
decoding the descriptors, we observe the two existing decoding strategies excel
in different areas, which are complementary. A Consistency-Aware Fusion (CAF)
strategy is proposed to leverage the advantages of both. We incorporate the
above ACA, SVA, and CAF into a coarse-to-fine framework, termed Geometry-aware
Reconstruction and Fusion-refined Rendering (GeFu). GeFu attains
state-of-the-art performance across multiple datasets. Code is available at
https://github.com/TQTQliu/GeFu .
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要