Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data

Abstract

Large scale databases with high-quality manual annotations are scarce in audio domain. We thus explore a self-supervised graph approach to learning audio representations from highly limited labeled data. Considering each audio sample as a graph node, we propose a subgraph-based framework with novel self-supervision tasks that can learn effective audio representations. During training, subgraphs are constructed by sampling the entire pool of available training data to exploit the relationship between the labeled and unlabeled audio samples. During inference, we use random edges to alleviate the overhead of graph construction. We evaluate our model on three benchmark audio databases, and two tasks: acoustic event detection and speech emotion recognition. Our semi-supervised model performs better or on par with fully supervised models and outperforms several competitive existing models. Our model is compact (240k parameters), and can produce generalized audio representations that are robust to different types of signal noise.

Visually-aware Acoustic Event Detection using Heterogeneous Graphs

“…We also compared our method with a graph-based work. Each node in this work represents an audio clip, and a KNN subgraph has been created, as well as a GNN that is trained using graph self-supervised proxy tasks [36]. We also use the two popular spatial and temporal network architectures, ResNet-1D [40] and LSTM, with pretrained embedding features for both audio and video as input, to further investigate the superiority of our graph modelling.…”

“…Our network weights are initialized following the Xavier initialization. We used Adam optimizer with a learning rate of 0.005, a decay rate of 0.1 after 1500 it- [35] 0.39 ± 0.02 -87M SSL graph [36] 0.42 ± 0.02 -218K Wave-Logmel [37] 0.43 ± 0.04 -81M AST [38] 0.44 ± 0.00 -88M…”