Multilingual Speech Evaluation: Case Studies on English, Malay and Tamil

Zhang, Huayun; Shi, Ke; Chen, Nancy F.

doi:10.21437/interspeech.2021-1258

Cited by 10 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Over the past few decades, extensive research has been conducted on spoken fluency scoring. Traditionally, handcrafted features such as the statistics of speech break [6], speech rate [6,7,[12][13][14], filled pause, and goodness of pronunciation (GOP) [7][8][9] were collected based on phone boundaries and fed into various fluency scorers such as SVM [12,14], and multiple linear [6]. Recent works have employed sequence models to directly learn utterance-level fluency scores from phonelevel raw features, including phonetic features (e.g., phone sequence [7][8][9][10]), prosodic features (e.g., energy [9], pitch [7] and phone duration [10].…”

Section: Related Workmentioning

confidence: 99%

“…Traditionally, handcrafted features such as the statistics of speech break [6], speech rate [6,7,[12][13][14], filled pause, and goodness of pronunciation (GOP) [7][8][9] were collected based on phone boundaries and fed into various fluency scorers such as SVM [12,14], and multiple linear [6]. Recent works have employed sequence models to directly learn utterance-level fluency scores from phonelevel raw features, including phonetic features (e.g., phone sequence [7][8][9][10]), prosodic features (e.g., energy [9], pitch [7] and phone duration [10]. Bi-directional Long Short Term Memory (BLSTM) [7,10,11,15] and Transformer models [8,9] have been used to capture the dynamic changes of phone-level pronunciation-related features for better modeling the evolution of local fluency over time.…”

Section: Related Workmentioning

confidence: 99%

“…Automatic scoring of fluency, serves as an essential module in computer-aided language learning (CALL) systems. It has been extensively studied in both "read aloud" [6][7][8][9][10][11] and "open response" [12][13][14][15] scenarios. In the read aloud" scenario, L2 learners are required to read a provided prompt text, whereas the 'open response" requires them to express their opinions freely based on a given question.…”

Section: Introductionmentioning

confidence: 99%

“…Fluency related features are then extracted and fed into subsequent fluency scorers. Although recent end-to-end neural network based fluency scorers have achieved satisfactory results [7][8][9][10][11]15], their performances heavily rely on the size of labeled scoring samples. In fact, the non-native data labeling process is costly and has scalability issues [16].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Using Fluency Representation Learned from Sequential Raw Features for Improving Non-native Fluency Scoring

Fu¹,

Gao²,

Tian³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

Speech fluency/disfluency can be evaluated by analyzing a range of phonetic and prosodic features. Deep neural networks are commonly trained to map fluency-related features into the human scores. However, the effectiveness of deep learning-based models is constrained by the limited amount of labeled training samples. To address this, we introduce a self-supervised learning (SSL) approach that takes into account phonetic and prosody awareness for fluency scoring. Specifically, we first pre-train the model using a reconstruction loss function, by masking phones and their durations jointly on a large amount of unlabeled speech and text prompts. We then fine-tune the pre-trained model using human-annotated scoring data. Our experimental results, conducted on datasets such as Speechocean762 and our non-native datasets, show that our proposed method outperforms the baseline systems in terms of Pearson correlation coefficients (PCC). Moreover, we also conduct an ablation study to better understand the contribution of phonetic and prosody factors during the pre-training stage.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Using Fluency Representation Learned from Sequential Raw Features for Improving Non-native Fluency Scoring

Fu¹,

Gao²,

Tian³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

show abstract

“…The word pronunciation score in each of them has been labeled by five experts, the median scores were adopted following the score files coming with the database, ranging from 0 to 10. Following the calculation method of previous study [30], the averaged inter-rater agreement is 0.726. In addition, linguistic experts in Bytedance collected a small amount of task-related unlabeled data (e.g., a group of Chinese adults are required to read aloud given English prompts).…”

Section: Experimental Setup 31 Speech Corpusmentioning

confidence: 99%

Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information

Fu¹,

Gao²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning-based pronunciation scoring models highly rely on the availability of the annotated non-native data, which is costly and has scalability issues. To deal with the data scarcity problem, data augmentation is commonly used for model pretraining. In this paper, we propose a phone-level mixup, a simple yet effective data augmentation method, to improve the performance of word-level pronunciation scoring. Specifically, given a phoneme sequence from lexicon, the artificial augmented word sample can be generated by randomly sampling from the corresponding phone-level features in training data, while the word score is the average of their GOP scores. Benefit from the arbitrary phone-level combination, the mixup is able to generate any word with various pronunciation scores. Moreover, we utilize multi-source information (e.g., MFCC and deep features) to further improve the scoring system performance. The experiments conducted on the Speechocean762 show that the proposed system outperforms the baseline by adding the mixup data for pretraining, with Pearson correlation coefficients (PCC) increasing from 0.567 to 0.61. The results also indicate that proposed method achieves similar performance by using 1/10 unlabeled data of baseline. In addition, the experimental results also demonstrate the efficiency of our proposed multisource approach.

show abstract