Category-Guided Localization Network for Visual Sound Source Separation.

Learning audiovisual correspondence plays an important role in understanding outside world. The object of this work is to clearly separate the audio corresponding to the given visual object from mixed audio, i.e., visually guided sound source separation. However, accurately obtaining and matching reference features in visual and audio modalities, which is crucial in separation tasks, still remains a tough challenge. In this paper, we propose a novel three-modality audio-visual separation framework, Category-Guided Localization Network (CGLN), to address this problem. The proposed category-guided approach can cover the sound object more precisely than others to obtain better separation. Object and motion features are used as guiding signals to realize sound source separation in an end-to-end manner Extensive experiments are performed on two publicly-available datasets, MUSIC and AudioSet. Our method has achieved the goal of modality matching and alignment, and the results could outperform other state-of-the-art approaches.