Self-supervised audio representation learning for mobile devices

Abstract

We explore self-supervised models that can be potentially deployed on mobile devices to learn general purpose audio representations. Specifically, we propose methods that exploit the temporal context in the spectrogram domain. One method estimates the temporal gap between two short audio segments extracted at random from the same audio clip. The other methods are inspired by Word2Vec, a popular technique used to learn word embeddings, and aim at reconstructing a temporal spectrogram slice from past and future slices or, alternatively, at reconstructing the context of surrounding slices from the current slice. We focus our evaluation on small encoder architectures, which can be potentially run on mobile devices during both inference (re-using a common learned representation across multiple downstream tasks) and training (capturing the true data distribution without compromising users' privacy when combined with federated learning). We evaluate the quality of the embeddings produced by the self-supervised learning models, and show that they can be re-used for a variety of downstream tasks, and for some tasks even approach the performance of fully supervised models of similar size.Preprint. Under review.

Sense and Learn: Self-Supervision for Omnipresent Sensors

“…For assessing the quality of the self-supervised embeddings, we conduct experiments with a linear classifier on the end-tasks. Linear separability is a standard way of measuring the power of self-supervised-learned features in the literature [12,38,52], i.e. if the representations disentangle factors of variations in the input, then it becomes easier to solve subsequent tasks.…”

“…A similar technique is proposed in [54] to learn from multiple views of the data. [52] defined self-supervised tasks for audio, inspired by word2vec [34]. [25] showed that video representations could be learned by exploiting audio-visual temporal synchronization.…”

“…We demonstrate that a relatively straightforward suite of auxiliary tasks results in meaningful features for diverse problems, including: activity recognition, stress detection, sleep stage scoring, and WiFi sensing. First, we show that the self-supervised representations are highly competitive with those learned with a fully-supervised model, by training a linear classifier on top of the frozen network, as it is a standard evaluation protocol for assessing the quality of self-supervised tasks [38,52]. Second, we explore fine-tuning the last layer of the encoder to gain further improvements over training from scratch.…”

Unsupervised Speech Representation Learning for Behavior Modeling using Triplet Enhanced Contextualized Networks

“…for downstream tasks. In speech representation learning (Latif et al, 2020), unsupervised techniques such as autoregressive modeling (Chung, Hsu, Tang and Glass, 2019;Chung and Glass, 2020a,b) and self-supervised modeling (Milde and Biemann, 2018;Tagliasacchi, Gfeller, Quitry and Roblek, 2019;Pascual, Ravanelli, Serrà, Bonafonte and Bengio, 2019) employ temporal context information for extracting speech representation. In our prior behavior modeling work, an unsupervised representative learning framework was proposed (Li, Baucom and Georgiou, 2017), which showed the promise of learning behavior representations based on the behavior stationarity hypothesis that nearby segments of speech share the same behavioral context.…”

Contrastive Learning of General-Purpose Audio Representations

“…Through utilizing a triplet loss as an unsupervised objective with a subset of AudioSet [14] for model training, they showed improved performance on several downstream speech classification tasks. Inspired from seminal work in NLP [15], the work in [16] adopted a similar approach to learn audio representations (i.e. AUDIO2VEC) along with another "pretext" task of estimating temporal distance between audio segments.…”