Pre-Training Audio Representations With Self-Supervision

Abstract

We explore self-supervision as a way to learn general purpose audio representations. Specifically, we propose two self-supervised tasks: Audio2Vec, which aims at reconstructing a spectrogram slice from past and future slices and Temporal-Gap, which estimates the distance between two short audio segments extracted at random from the same audio clip. We evaluate how the representations learned via self-supervision transfer to different downstream tasks, either training a task-specific linear classifier on top of the pretrained embeddings, or fine-tuning a model end-to-end for each downstream task. Our results show that the representations learned with Audio2Vec transfer better than those learned by fully-supervised training on Audioset. In addition, by fine-tuning Audio2Vec representations it is possible to outperform fully-supervised models trained from scratch on each task, when limited data is available, thus improving label efficiency.

Contrastive Learning of General-Purpose Audio Representations

“…We then compare COLA to prior self-supervised methods proposed in [16,25], including a standard triplet loss, AU-DIO2VEC (CBoW and SG) and temporal gap prediction models. Here, the CBoW and SG are generative models inspired from WORD2VEC, trained to reconstruct a randomly selected temporal slice of log-mel spectrograms given the rest or vice versa.…”

“…CBoW [16,25] SG [16,25] TemporalGap [16,25] Triplet Loss [16,25] TRILL [13] ble 2 shows that COLA embeddings consistently outperform all these methods. In particular, on acoustic scene classification, we obtain a competitive accuracy of 94% compared to 73% achieved with a triplet loss in [16].…”

“…It contains 2 millions excerpts of 10 seconds audio from YouTube videos that are annotated in a multi-label fashion with over 500 classes. This dataset has been used by [16,25,13] for self-supervised pre-training. Since our method is self-supervised, we never use Audioset labels.…”

“…We perform downstream evaluation on a variety of tasks, including both speech and non-speech. To allow for comparison with previous methods, we rely on datasets that have been previously used by [16,25,13]. For speaker identification, we use a 100-hours subset of LibriSpeech (LBS) [26] that contains audio of books read by 251 speakers, as well as the Voxceleb [27] subset used in [13], with 1, 251 speakers.…”

Unsupervised Contrastive Learning of Sound Event Representations

“…These representations can then be used for downstream tasks, for example where only few data, or poorly labeled data, are available. Self-supervised learning has shown great promise in computer vision [9,10], and interesting results in the audio domain [7,11]. One of the first works in self-supervised sound event representation learning is [7], adopting a triplet loss-based training by creating anchor-positive pairs via simple audio transformations, e.g., adding noise or mixing examples.…”