Non-Contrastive Self-Supervised Learning for Utterance-Level Information Extraction From Speech

Abstract

In recent studies, self-supervised pre-trained models tend to outperform supervised pre-trained models in transfer learning. In particular, self-supervised learning of utterance-level speech representation can be used in speech applications that require discriminative representation of consistent attributes within an utterance: speaker, language, emotion, and age. Existing frame-level self-supervised speech representation, e.g., wav2vec, can be used as utterance-level representation with pooling, but the models are usually large. There are also self-supervised learning techniques to learn utterance-level representation. One of the most successful is a contrastive method, which requires negative sampling: selecting alternative samples to contrast with the current sample (anchor). However, this does not ensure that all the negative samples belong to classes different from the anchor class without labels. This paper applies a non-contrastive self-supervised method to learn utterance-level embeddings. We adapted DIstillation with NO labels (DINO) from computer vision to speech. Unlike contrastive methods, DINO does not require negative sampling. We compared DINO to x-vector trained in a supervised manner. When transferred to downstream tasks (speaker verification, speech emotion recognition, and Alzheimer's disease detection), DINO outperformed x-vector. We studied the influence of several aspects during transfer learning such as dividing the fine-tuning process into steps, chunk lengths, or augmentation. During fine-tuning, tuning the last affine layers first and then the whole network surpassed fine-tuning all at once. Using shorter chunk lengths, although they generate more diverse inputs, did not necessarily improve performance, implying speech segments at least with a specific length are required for better performance per application. Augmentation was helpful in speech emotion recognition.

Self-Supervised Training of Speaker Encoder With Multi-Modal Diverse Positive Pairs

“…II. For speaker verification, MCL-DPP achieves an EER of 2.89%, 3.34% and 6.47% on Vox-O, Vox-E and Vox-H, respectively, that outperform the best prior work, i.e., Cho et al [40] by 40.17% on Vox-O. For face verification, it also achieves an EER of 1.74% in Vox-O.…”

“…Other comparison-based self-supervised learning techniques include the MOCO framework [38], [39], which stores the negative pairs in the memory bank; the DINO framework [12], [40]- [42] that only involves positive pairs and achieves considerable improvement. For efficiency and effectiveness, we adopt the SCL framework in this study and focus on the sampling strategy of positive pairs.…”

“…We use a small labelled validation set to monitor the progressive clustering process, which is valid and commonly used in related studies [15], [39], [40], [50] and the competition 12 .…”

An AI blue team playbook

“…These defenses can fall under the following categories: detect and remove the attack vector; [28][29][30] implement non-differentiable functions to obscure gradients; 31,32 sanitize the attack vector to eliminate adversarial perturbations; [33][34][35] and apply formal verification [36][37][38] or certification techniques [39][40][41] to provide performance guarantees. Defenses applied to training data protect against poisoning attacks by filtering out potentially poisoned data samples [42][43][44][45][46] . Defenses within the training algorithm employ robust training techniques, such as adversarial Table 2.…”

Label-Efficient Self-Supervised Speaker Verification With Information Maximization and Contrastive Learning

“…Recently, motivated by the surge of self-supervised learning concepts, many deep embedding methods [7,8,9,10,11] The code associated with this article is publicly available at https://github.com/theolepage/sslsv. have proven to be very effective in benefiting from the massive amount of unlabeled data.…”