Self-Supervised Speech Representation Learning: A Review

Mohamed, Abdelrahman; Lee, Hung-yi; Borgholt, Lasse; Havtorn, Jakob D.; Edin, Joakim; Igel, Christian; Kirchhoff, Katrin; Li, Shang-Wen; Livescu, Karen; Maaløe, Lars; Sainath, Tara N.; Watanabe, Shinji

doi:10.1109/jstsp.2022.3207050

Cited by 144 publications

(45 citation statements)

References 286 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We measure the extent of acoustic, phonetic, and word content encoded in individual layers for 11 pre-trained models, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that phonetic and word information concentrates in different layers 1 The codebase will be made available at: https://github.com/ankitapasad/layerwise-analysis/ for different models, and the layer-wise trends relate to the pre-training objectives, despite differences in training data. We also report CCA measurements for a randomly initialized model and note that there is no inductive bias of the model architecture that affects the observed trends.…”

Section: Introductionmentioning

confidence: 99%

“…Self-supervised models have become a nearly ubiquitous approach for learning speech representations and improving performance on downstream tasks [1][2][3][4][5], but our understanding of their properties and strategies for their use is still limited. Some recent work has begun developing an understanding of the extent and location of different acoustic and linguistic information encoded by these models [6][7][8][9][10], which in some cases has resulted in improved fine-tuning strategies [8,9].…”

Section: Introductionmentioning

confidence: 99%

“…Self-supervised networks are trained with an objective function that solves a pretext task formulated using unlabeled data. Many self-supervised speech representation models have been proposed in the last few years [1]. We present analysis for eleven speech models differing in (i) training objective, (ii) training data modality (using either only speech or image-speech pairs), (iii) training data languages (English or multilingual), and (iv) model size.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Layer-Wise Analysis of a Self-Supervised Speech Representation Model

Pasad

Chou

Livescu

2021

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

114

View full text Add to dashboard Cite

Many self-supervised speech models, varying in their pretraining objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive empirical successes on downstream tasks, we still have a limited understanding of the properties encoded by the models and the differences across models. In this work, we examine the intermediate representations for a variety of recent models. Specifically, we measure acoustic, phonetic, and word-level properties encoded in individual layers, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that these properties evolve across layers differently depending on the model, and the variations relate to the choice of pre-training objective. We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks. We discover that CCA trends provide reliable guidance to choose layers of interest for downstream tasks and that single-layer performance often matches or improves upon using all layers, suggesting implications for more efficient use of pre-trained models. 1

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Layer-Wise Analysis of a Self-Supervised Speech Representation Model

Pasad

Chou

Livescu

2021

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

114

View full text Add to dashboard Cite

show abstract

“…Wave; JSTSP Feb. 2022 300-306 Ma, J., see Wang, H., JSTSP Feb. 2022 276-288 Ma, M., see 1519-1532 Ma, Z., see Guo, P., JSTSP June 2022 828-842 Ma, Z., see Huang, E., JSTSP June 2022 608-621 Maaloe, L., see Mohamed, A., JSTSP Oct. 20221179-1210 Malkin, E., see Ponomarchuk, A., JSTSP Feb. 2022 175-187 Mangia, M., see Martinini, F., JSTSP June 2022 713-724 Mao, S., Ji, M., Wang, B., Dai, Q., and Fang, L., Lee, H., Borgholt, L., Havtorn, J.D., Edin, J., Igel, C., Kirchhoff, K., Li, S., Livescu, K., Maaloe, L., Sainath, T.N., and Watanabe, S., Self-Supervised Speech Representation Learning: A Review; JSTSP Oct. 2022 1179-1210 Mohamed, A., see 1174-1178 Moreno, P., see Baskar, M.K., 1357-1366 Mori, K., see 1480-1492 Moritz, N., see Higuchi, Y., 1424-1438 Moro-Velazquez, L., see Cho, J., 1284-1295 Mu, X., see Xu, J., JSTSP JSTSP Jan. 2022 7-25 Poor, H.V., see Kim, K.J., JSTSP Jan. 2022 2-6 Poor, H.V., see Yang, H.H., JSTSP April 2022 406-419 Poor, H.V., see Wang, S., JSTSP April 2022 501-515 Poor, H.V., see Nguyen, K.K., JSTSP Aug. 1086Aug.…”

Section: Mk a Mathematical Modeling Of Covid-19 And Prediction Of Upc...mentioning

confidence: 99%

2022 Index IEEE Journal of Selected Topics in Signal Processing Vol. 16

2022

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

This index covers all technical items-papers, correspondence, reviews, etc.-that appeared in this periodical during 2022, and items from previous years that were commented upon or corrected in 2022. Departments and other items may also be covered if they have been judged to have archival value.The Author Index contains the primary entry for each item, listed under the first author's name. The primary entry includes the coauthors' names, the title of the paper or other item, and its location, specified by the publication abbreviation, year, month, and inclusive pagination. The Subject Index contains entries describing the item under all appropriate subject headings, plus the first author's name, the publication abbreviation, month, and year, and inclusive pages. Note that the item title is found only under the primary entry in the Author Index.

show abstract

“…SSL pre-trains a shared representation model on a huge amount of unlabeled data. The pre-trained SSL model can be used for various downstream tasks with minimal adaptation via either finetuning or utilizing the learned representation from the frozen model [1]. Applying a SSL model to different downstream tasks can significantly lower the entry barrier for developing a model compared to training the model from scratch.…”

Section: Introductionmentioning

confidence: 99%

Exploring Efficient-tuning Methods in Self-supervised Speech Models

Chen¹,

Fu²,

Chih-Ying³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this study, we aim to explore efficient tuning methods for speech self-supervised learning. Recent studies show that self-supervised learning (SSL) can learn powerful representations for different speech tasks. However, fine-tuning pre-trained models for each downstream task is parameterinefficient since SSL models are notoriously large with millions of parameters. Adapters are lightweight modules commonly used in NLP to solve this problem. In downstream tasks, the parameters of SSL models are frozen, and only the adapters are trained. Given the lack of studies generally exploring the effectiveness of adapters for self-supervised speech tasks, we intend to fill this gap by adding various adapter modules in pre-trained speech SSL models. We show that the performance parity can be achieved with over 90% parameter reduction, and discussed the pros and cons of efficient tuning techniques. This is the first comprehensive investigation of various adapter types across speech tasks.

show abstract

Self-Supervised Speech Representation Learning: A Review

Cited by 144 publications

References 286 publications

Layer-Wise Analysis of a Self-Supervised Speech Representation Model

Layer-Wise Analysis of a Self-Supervised Speech Representation Model

2022 Index IEEE Journal of Selected Topics in Signal Processing Vol. 16

Exploring Efficient-tuning Methods in Self-supervised Speech Models

Contact Info

Product

Resources

About