A WAV2VEC2-Based Experimental Study on Self-Supervised Learning Methods to Improve Child Speech Recognition

Abstract

Despite recent advancements in deep learning technologies, Child Speech Recognition remains a challenging task. Current Automatic Speech Recognition (ASR) models require substantial amounts of annotated data for training, which is scarce. In this work, we explore using the ASR model, wav2vec2, with different pretraining and finetuning configurations for self-supervised learning (SSL) toward improving automatic child speech recognition. The pretrained wav2vec2 models were finetuned using different amounts of child speech training data, adult speech data, and a combination of both, to discover the optimum amount of data required to finetune the model for the task of child ASR. Our trained model achieves the best Word Error Rate (WER) of 7.42 on the MyST child speech dataset, 2.91 on the PFSTAR dataset and 12.77 on the CMU KIDS dataset using cleaned variants of each dataset. Our models outperformed the unmodified wav2vec2 BASE 960 on child speech using as little as 10 hours of child speech data in finetuning. The analysis of different types of training data and their effect on inference is provided by using a combination of custom datasets in pretraining, finetuning and inference. These 'cleaned' datasets are provided for use by other researchers to provide comparisons with our results.

MDVC corpus: empowering Moroccan Darija speech recognition

“…Central to its success is the concept of selfsupervised learning, a paradigm shift in ASR research. By training on vast amounts of unlabeled speech data [21], Wav2Vec2 transcends the limitations of traditional supervised methods. This innovation not only significantly boosts data efficiency but also empowers the model to grasp the intricacies of diverse accents and languages, making it a versatile tool for multilingual ASR applications.…”

Exploration of Whisper fine-tuning strategies for low-resource ASR

“…Well-known self-supervised speech models such as Wav2vec series [19,20], Hubert [21], WavLM [22], and MMS (Massively Multilingual Speech) [23] require fine-tuning on domain-specific data to adapt to downstream tasks. Jain et al explored different pretraining and fine-tuning methods for the Wav2vec 2.0 model on the ASR task for child speech [24]. Zhang et al analyzed various combinations of pretraining and fine-tuning on 15 low-resource languages in the OpenASR21 challenge [25].…”

Augmentation Techniques for Adult-Speech to Generate Child-Like Speech Data Samples at Scale

“…• Original_220h: Contains 220 hours of original adult speech. • MyST_55h: Contains 55 hours of cleaned MyST child speech, which was prepared according to [56] The Original_12h and Original_220h sets are the original Librispeech (adult speech) counterparts of the Augmented_17h and Augmented_311h sets, respectively. Note that there is an increase in the number of hours of speech data when augmenting from Original_12h to Augmented_17h and from Original_220h to Augmented_311h, in both cases by a factor of 1.41.…”

“…We used four test datasets to test our finetuned models at the inference stage. These datasets were prepared in accordance with our previous research on child speech ASR [56]. Since MyST [30] is the largest child audio corpus available publicly for research use, it was used for both finetuning and inference.…”