Automatic Speech Fluency Evaluation Method Based on Multimodality for Putonghua Proficiency Test Propositional Speaking

Abstract

The Putonghua Proficiency Test (Putonghua Shuiping Ceshi, PSC) is a speaking test in China. The propositional speaking section in PSC focuses on the ability of speakers to express ideas fluently and accurately without textual reference. However, unlike other sections of the PSC, propositional speaking is still scored manually, which can result in inefficiency, high costs, and subjectivity. To address these issues, an automatic speech fluency evaluation method based on multimodality is proposed. First, different neural networks are used to extract unimodal features. Then, cross-modal attention is applied to achieve multimodal fusion. Finally, fluency evaluation results are obtained using self-attention to reinforce high-contributing information. The proposed method achieves 81.67[Formula: see text] accuracy on a self-built dataset, demonstrating that combining textual and acoustic features provides complementary information to improve automatic speech fluency evaluation accuracy.