SIFNet: Free-form image inpainting using color split-inpaint-fuse approach
Computer Vision and Image Understanding(2022)
Abstract
Recent deep learning-based approaches have shown outstanding performance in generating visually plausible and refined contents for the missing regions in free-form image inpainting tasks. However, most of the existing methods employ a coarse-to-refine approach where the refinement process depends on a single coarse estimation, often leading to texture and structure inconsistencies. Though several existing methods focus on incorporating additional inputs to mitigate this problem, no learning-based studies have investigated the effects of decomposing input corrupted image into luma and chroma images and performing decoupled inpainting of the decomposed components. To this end, we propose a Split-Inpaint-Fuse Network (SIFNet), an end-to-end two-stage inpainting approach that uses a split-inpaint sub-network for separately inpainting the corrupted luma and chroma images using two decoupled branches in the coarse stage and a fusion sub-network for fusing the inpainted luma and chroma images into a refined image in the refinement stage. Additionally, we propose two attention mechanisms for the coarse stage - a progressive context module to find the patch-level feature similarity for the luma image reconstruction and a spatial-channel context module to find important spatial and channel features for the chroma image reconstruction. Experimental results reveal that our Split-Inpaint-Fuse approach outperforms the existing inpainting methods by comparative margins. In addition, extensive ablation studies confirm the effectiveness of the proposed approach, constituting modules and architectural choices.
MoreTranslated text
Key words
Image inpainting,Convolutional neural network,Generative adversarial networks,Attention mechanisms,Color space decomposition
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined