Audio Self-supervised Learning: A Survey

Liu, Shuo; Mallol-Ragolta, Adria; Parada-Cabeleiro, Emilia; Qian, Kun; Jing, Xin; Kathan, Alexander; Hu, Bin; Schuller, Björn

doi:10.48550/arxiv.2203.01205

Cited by 3 publications

(4 citation statements)

References 93 publications

(168 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As acoustic features we use: 1. total signal duration, 2. zero crossing rate, 3. mean pitch, 4. local jitter, 5. local shimmer, 6. energy entropy, 7. spectral centroid, and 8. voiced to unvoiced ratio. The acoustic features are extracted using the ComParE2016 [30] feature set of our openSMILE toolkit 5 [31] -with the exception of duration, which is obtained with audiofile. 6 We evaluate predictive performance using root mean squared error (RMSE).…”

Section: Probing #3: Feature Probingmentioning

confidence: 99%

“…Recently, deep neural networks (DNNs) consisting of selfattention layers (i. e., transformers) have provided state-of-theart results for speech emotion recognition (SER) and have substantially improved valence prediction [1,2,3,4,5]. These models are typically pre-trained on large corpora in a self-supervised fashion, with the main goal of improving automatic speech recognition performance; thus, they capture a large amount of linguistic information that is beneficial for that task.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Probing Speech Emotion Recognition Transformers for Linguistic Knowledge

Triantafyllopoulos¹,

Wagner²,

Wierstorf³

et al. 2022

Preprint

View full text Add to dashboard Cite

Large, pre-trained neural networks consisting of self-attention layers (transformers) have recently achieved state-of-the-art results on several speech emotion recognition (SER) datasets. These models are typically pre-trained in self-supervised manner with the goal to improve automatic speech recognition performance -and thus, to understand linguistic information. In this work, we investigate the extent in which this information is exploited during SER fine-tuning. Using a reproducible methodology based on open-source tools, we synthesise prosodically neutral speech utterances while varying the sentiment of the text. Valence predictions of the transformer model are very reactive to positive and negative sentiment content, as well as negations, but not to intensifiers or reducers, while none of those linguistic features impact arousal or dominance. These findings show that transformers can successfully leverage linguistic information to improve their valence predictions, and that linguistic analysis should be included in their testing.

show abstract

Section: Probing #3: Feature Probingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Probing Speech Emotion Recognition Transformers for Linguistic Knowledge

Triantafyllopoulos¹,

Wagner²,

Wierstorf³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…There are methods such as TERA (Liu et al, 2021a), Conformer (Gulati et al, 2020) (convolution-augmented transformers, used in speech recognition), and then ViT-like approaches such as the Keyword Transformer (KWT) (Berg et al, 2021) and the Audio Spectrogram Transformer (AST) (Gong et al, 2021a). In recent self-supervised audio representation learning methods, transformer-based encoders have seen much use alongside convolutional or convolutionalrecurrent encoders (Liu et al, 2022).…”

Section: Transformersmentioning

confidence: 99%

Learning Audio Representations with MLPs

Morshed¹,

Ahsan²,

Mahmud³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose an efficient MLP-based approach for learning audio representations, namely timestamp and scene-level audio embeddings. We use an encoder consisting of sequentially stacked gated MLP blocks, which accept 2D MFCCs as inputs. In addition, we also provide a simple temporal interpolation-based algorithm for computing scene-level embeddings from timestamp embeddings. The audio representations generated by our method are evaluated across a diverse set of benchmarks at the Holistic Evaluation of Audio Representations (HEAR) challenge, hosted at the NeurIPS 2021 competition track. We achieved first place on the Speech Commands (full), Speech Commands (5 hours), and the Mridingham Tonic benchmarks. Furthermore, our approach is also the most resource-efficient among all the submitted methods, in terms of both the number of model parameters and the time required to compute embeddings.

show abstract

“…Attention mechanisms significantly impact deep learning models in many fields, which enrich the information the model could learn from inputs [15]. Attention mechanisms can select, modulate, and focus on the information most important to the target of our problem, like human attention [10]. Therefore, this paper will investigate the effectiveness of data augmentation, SSL models, and attention modules on emotion recognition via vocal burst.…”

Section: Introductionmentioning

confidence: 99%

Self-Relation Attention and Temporal Awareness for Emotion Recognition via Vocal Burst

Trinh¹,

Vo²,

Lee³

2022

Preprint

View full text Add to dashboard Cite

The technical report presents our emotion recognition pipeline for high-dimensional emotion task (A-VB High) in The ACII Affective Vocal Bursts (A-VB) 2022 Workshop & Competition. Our proposed method contains three stages. Firstly, we extract the latent features from the raw audio signal and its Mel-spectrogram by self-supervised learning methods. Then, the features from the raw signal are fed to the self-relation attention and temporal awareness (SA-TA) module for learning the valuable information between these latent features. Finally, we concatenate all the features and utilize a fully-connected layer to predict each emotion's score. By empirical experiments, our proposed method achieves a mean concordance correlation coefficient (CCC) of 0.7295 on the test set, compared to 0.5686 on the baseline model. The code of our method is available at https://github.com/linhtd812/A-VB2022.

show abstract

Audio Self-supervised Learning: A Survey

Cited by 3 publications

References 93 publications

Probing Speech Emotion Recognition Transformers for Linguistic Knowledge

Probing Speech Emotion Recognition Transformers for Linguistic Knowledge

Learning Audio Representations with MLPs

Self-Relation Attention and Temporal Awareness for Emotion Recognition via Vocal Burst

Contact Info

Product

Resources

About