Automatic Fluency Assessment Method for Spontaneous Speech without Reference Text

Abstract

The automatic fluency assessment of spontaneous speech without reference text is a challenging task that heavily depends on the accuracy of automatic speech recognition (ASR). Considering this scenario, it is necessary to explore an assessment method that combines ASR. This is mainly due to the fact that in addition to acoustic features being essential for assessment, the text features output by ASR may also contain potentially fluency information. However, most existing studies on automatic fluency assessment of spontaneous speech are based solely on audio features, without utilizing textual information, which may lead to a limited understanding of fluency features. To address this, we propose a multimodal automatic speech fluency assessment method that combines ASR output. Specifically, we first explore the relevance of the fluency assessment task to the ASR task and fine-tune the Wav2Vec2.0 model using multi-task learning to jointly optimize the ASR task and fluency assessment task, resulting in both the fluency assessment results and the ASR output. Then, the text features and audio features obtained from the fine-tuned model are fed into the multimodal fluency assessment model, using attention mechanisms to obtain more reliable assessment results. Finally, experiments on the PSCPSF and Speechocean762 dataset suggest that our proposed method performs well in different assessment scenarios.

Automatic Speech Disfluency Detection Using wav2vec2.0 for Different Languages with Variable Lengths

“…PSC-PS-DF consists of propositional speaking files for the Mandarin Proficiency Test (Putonghua Shuiping Ceshi, PSC). Previous work has shown that, due to the nature of Chinese propositional speaking, which requires the speaker to freely describe a topic for three minutes without any reference text, the speech files contain a large number of disfluent features, such as "um", "ah", and "uh" interjections, blocks, prolongations, and repetition, but such disfluent features are rarely marked and used in research [55]. In this study, disfluent features were annotated in Chinese propositional speaking data to obtain spontaneously spoken disfluent features in Chinese for the detection of disfluent Chinese-language speech.…”