Positive and negative sampling strategies for self-supervised learning on audio-video data
arxiv(2024)
摘要
In Self-Supervised Learning (SSL), Audio-Visual Correspondence (AVC) is a
popular task to learn deep audio and video features from large unlabeled
datasets. The key step in AVC is to randomly sample audio and video clips from
the dataset and learn to minimize the feature distance between the positive
pairs (corresponding audio-video pair) while maximizing the distance between
the negative pairs (non-corresponding audio-video pairs). The learnt features
are shown to be effective on various downstream tasks. However, these methods
achieve subpar performance when the size of the dataset is rather small. In
this paper, we investigate the effect of utilizing class label information in
the AVC feature learning task. We modified various positive and negative data
sampling techniques of SSL based on class label information to investigate the
effect on the feature quality. We propose a new sampling approach which we call
soft-positive sampling, where the positive pair for one audio sample is not
from the exact corresponding video, but from a video of the same class.
Experimental results suggest that when the dataset size is small in SSL setup,
features learnt through the soft-positive sampling method significantly
outperform those from the traditional SSL sampling approaches. This trend holds
in both in-domain and out-of-domain downstream tasks, and even outperforms
supervised classification. Finally, experiments show that class label
information can easily be obtained using a publicly available classifier
network and then can be used to boost the SSL performance without adding extra
data annotation burden.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要