WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

Chen, Sanyuan; Wang, Chengyi; Chen, Zhengyang; Wu, Yu; Liu, Shujie; Chen, Zhuo; Li, Jinyu; Kanda, Naoyuki; Yoshioka, Takuya; Xiao, Xiong; Wu, Jian; Zhang, Long; Ren, Shuo; Ye, Qian; Zeng, Michael; Wei, Furu

doi:10.1109/jstsp.2022.3188113

Cited by 365 publications

(158 citation statements)

References 79 publications

Supporting

Mentioning

143

Contrasting

Order By: Relevance

“…Such pre-trained features can help to mitigate the small amount of overlapped speech data, because they were learnt using huge quantity of data. Our work is based on the new feature extractor created by Microsoft, named WavLM [19]. This system is a new self-supervised system built with transformer blocks trained on Mix94k, a corpus of 94k hours drawn from LibriLight, VoxPopuli and GigaSpeech.…”

Section: Pre-trained Features For Audio Segmentationmentioning

confidence: 99%

See 1 more Smart Citation

Overlapped speech and gender detection with WavLM pre-trained features

Lebourdais¹,

Tahon²,

Laurent³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

This article focuses on overlapped speech and gender detection in order to study interactions between women and men in French audiovisual media (Gender Equality Monitoring project). In this application context, we need to automatically segment the speech signal according to speakers gender, and to identify when at least two speakers speak at the same time. We propose to use WavLM model which has the advantage of being pre-trained on a huge amount of speech data, to build an overlapped speech detection (OSD) and a gender detection (GD) systems. In this study, we use two different corpora. The DIHARD III corpus which is well adapted for the OSD task but lack gender information. The ALLIES corpus fits with the project application context. Our best OSD system is a Temporal Convolutional Network (TCN) with WavLM pre-trained features as input, which reaches a new state-of-the-art F1-score performance on DIHARD. A neural GD is trained with WavLM inputs on a gender balanced subset of the French broadcast news ALLIES data, and obtains an accuracy of 94.9%. This work opens new perspectives for human science researchers regarding the differences of representation between women and men in French media.

show abstract

Section: Pre-trained Features For Audio Segmentationmentioning

confidence: 99%

“…WavLM: The second set of features is extracted with WavLM [19]. For OSD, we used the large version of WavLM 3 which returns 1024-dimension vectors per frame, without finetuning the model.…”

Section: Featuresmentioning

confidence: 99%

Overlapped speech and gender detection with WavLM pre-trained features

Lebourdais¹,

Tahon²,

Laurent³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

show abstract

“…The field of unsupervised representation learning is established enough that it is no longer necessary to channel it through special events. Indeed, self-supervised audio models are such an active domain that there are many relevant new models (for example, WavLM: [125]) which have yet to be evaluated on the ZRC metrics. Existing benchmarks, especially for Tasks 2 and 4, also still have a lot of potential for improvement, without creating more difficult tasks.…”

Section: The Future Of the Zero Resource Speech Challengementioning

confidence: 99%

Self-Supervised Language Learning From Raw Audio: Lessons From the Zero Resource Speech Challenge

Dunbar

Hamilakis²,

Dupoux³

2022

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

Recent progress in self-supervised or unsupervised machine learning has opened the possibility of building a full speech processing system from raw audio without using any textual representations or expert labels such as phonemes, dictionaries or parse trees. The contribution of the Zero Resource Speech Challenge series since 2015 has been to break down this long-term objective into four well-defined tasks-Acoustic Unit Discovery, Spoken Term Discovery, Discrete Resynthesis, and Spoken Language Modeling-and introduce associated metrics and benchmarks enabling model comparison and cumulative progress. We present an overview of the six editions of this challenge series since 2015, discuss the lessons learned, and outline the areas which need more work or give puzzling results.

show abstract

“…For deriving SSL embeddings, we make use of the following publicly available state-of-the-art (SOTA) pre-trained SSL systems like: Wav2Vec2 (Baevski et al, 2020), HuBERT (Hsu et al, 2021), and WavLM (Chen et al, 2021). These systems are among the top three performing networks for the SUPERB challenge (Yang et al, 2021), a SSL benchmark challenge for the speech processing tasks.…”

Section: Extracting Neural Embedding-based Fixed-length Feature Repre...mentioning

confidence: 99%

Comparing supervised and self-supervised embedding for ExVo Multi-Task learning track

Purohit¹,

Mahmoud²,

Vlasenko³

et al. 2022

Preprint

View full text Add to dashboard Cite

The ICML Expressive Vocalizations (ExVo) Multi-task challenge 2022, focuses on understanding the emotional facets of the non-linguistic vocalizations (vocal bursts (VB)). The objective of this challenge is to predict emotional intensities for VB, being a multi-task challenge it also requires to predict speakers' age and native-country. For this challenge we study and compare two distinct embedding spaces namely, self-supervised learning (SSL) based embeddings and task-specific supervised learning based embeddings. Towards that, we investigate feature representations obtained from several pre-trained SSL neural networks and task-specific supervised classification neural networks. Our studies show that the best performance is obtained with a hybrid approach, where predictions derived via both SSL and task-specific supervised learning are used. Our best system on test-set surpasses the ComPARE baseline (harmonic mean of all sub-task scores i.e.,) by a relative 13% margin.

show abstract

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

Cited by 365 publications

References 79 publications

Overlapped speech and gender detection with WavLM pre-trained features

Overlapped speech and gender detection with WavLM pre-trained features

Self-Supervised Language Learning From Raw Audio: Lessons From the Zero Resource Speech Challenge

Comparing supervised and self-supervised embedding for ExVo Multi-Task learning track

Contact Info

Product

Resources

About