Audio-Infused Automatic Image Colorization by Exploiting Audio Scene Semantics
CoRR(2024)
Abstract
Automatic image colorization is inherently an ill-posed problem with
uncertainty, which requires an accurate semantic understanding of scenes to
estimate reasonable colors for grayscale images. Although recent
interaction-based methods have achieved impressive performance, it is still a
very difficult task to infer realistic and accurate colors for automatic
colorization. To reduce the difficulty of semantic understanding of grayscale
scenes, this paper tries to utilize corresponding audio, which naturally
contains extra semantic information about the same scene. Specifically, a novel
audio-infused automatic image colorization (AIAIC) network is proposed, which
consists of three stages. First, we take color image semantics as a bridge and
pretrain a colorization network guided by color image semantics. Second, the
natural co-occurrence of audio and video is utilized to learn the color
semantic correlations between audio and visual scenes. Third, the implicit
audio semantic representation is fed into the pretrained network to finally
realize the audio-guided colorization. The whole process is trained in a
self-supervised manner without human annotation. In addition, an audiovisual
colorization dataset is established for training and testing. Experiments
demonstrate that audio guidance can effectively improve the performance of
automatic colorization, especially for some scenes that are difficult to
understand only from visual modality.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined