Similarity Analysis of Self-Supervised Speech Representations

Abstract

Self-supervised speech representation learning has recently been a prosperous research topic. Many algorithms have been proposed for learning useful representations from large-scale unlabeled data, and their applications to a wide range of speech tasks have also been investigated. However, there has been little research focusing on understanding the properties of existing approaches. In this work, we aim to provide a comparative study of some of the most representative self-supervised algorithms. Specifically, we quantify the similarities between different self-supervised representations using existing similarity measures. We also design probing tasks to study the correlation between the models' pretraining loss and the amount of specific speech information contained in their learned representations. In addition to showing how various self-supervised models behave differently given the same input, our study also finds that the training objective has a higher impact on representation similarity than architectural choices such as building blocks (RNN/Transformer/CNN) and directionality (uni/bidirectional). Our results also suggest that there exists a strong correlation between pre-training loss and downstream performance for some self-supervised algorithms.

Assessing the State of Self-Supervised Human Activity Recognition using Wearables

“…They observed that standard design recipes do not translate directly from end-to-end training to selfsupervision with the number of filters being a significant factor. Finally, some approaches for post-hoc analysis of the learned representations include studying the similarity of the learned representations to the gold standard that is supervised learning [10,20,25], and understanding the intrinsic dimensionality of the representations [23,71].…”

“…( 9) Masked reconstruction has the highest implicit dimensionality and thus more efficiently makes use of the learned representation space. (10) Utilizing the means and variances of the source dataset normalization on the target dataset results in considerable performance gains.…”

Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation

“…A growing number of self-supervised speech models have been proposed. Examples include contrastive predictive coding (CPC) [16,29], auto-regressive predictive coding [30], wav2vec [31], HuBERT [32,33], wav2vec 2.0 [12,34] and Wavlm [35], with all showing promising results for a variety of different speech processing tasks. Two particularly popular approaches, HuBERT and wav2vec 2.0, have been applied to automatic speech recognition [12,13], mispronunciation detection [36,37], speaker recognition [38,39] and emotion recognition [40].…”

A Practical Guide to Logical Access Voice Presentation Attack Detection

“…One advantage is that a large amount of data can be used even if they do not have target labels. This is a popular topic in speech processing community, and many models have been proposed: wav2vec [77], wav2vec2 [4], VQ-wav2vec [3], contrastive predictive coding [19], auto-regressive predictive coding [19], and HuBERT [36]. Readers are encouraged to check the related references and other papers in major conferences (e.g., ICASSP and Interspeech special sessions, and NISP workshop 6 ).…”