An ASR-free Fluency Scoring Approach with Self-Supervised Learning

Abstract

A typical fluency scoring system generally relies on an automatic speech recognition (ASR) system to obtain time stamps in input speech for either the subsequent calculation of fluency-related features or directly modeling speech fluency with an end-to-end approach. This paper describes a novel ASR-free approach for automatic fluency assessment using self-supervised learning (SSL). Specifically, wav2vec2.0 is used to extract frame-level speech features, followed by K-means clustering to assign a pseudo label (cluster index) to each frame. A BLSTM-based model is trained to predict an utterance-level fluency score from frame-level SSL features and the corresponding cluster indexes. Neither speech transcription nor time stamp information is required in the proposed system. It is ASR-free and can potentially avoid the ASR errors effect in practice. Experimental results carried out on non-native English databases show that the proposed approach significantly improves the performance in the "open response" scenario as compared to previous methods and matches the recently reported performance in the "read aloud" scenario.

Using Fluency Representation Learned from Sequential Raw Features for Improving Non-native Fluency Scoring

“…Recent works have employed sequence models to directly learn utterance-level fluency scores from phonelevel raw features, including phonetic features (e.g., phone sequence [7][8][9][10]), prosodic features (e.g., energy [9], pitch [7] and phone duration [10]. Bi-directional Long Short Term Memory (BLSTM) [7,10,11,15] and Transformer models [8,9] have been used to capture the dynamic changes of phone-level pronunciation-related features for better modeling the evolution of local fluency over time.…”

“…More recently, self-supervised learning (SSL)-based speech models such as wav2vec2 [25] have been shown to be effective in learning meaningful representations from raw speech signals in various downstream tasks [26]. Inspired by this success, researchers used pre-trained SSL models like wav2vec2 [25], Hu-BERT [27], and WavLM [28] to extract features directly and feed them into fluency scorers [9,11,15]. Due to the promising performance, we consider the two SSL-based models [9,15] as strong baselines of this work.…”

“…Inspired by this success, researchers used pre-trained SSL models like wav2vec2 [25], Hu-BERT [27], and WavLM [28] to extract features directly and feed them into fluency scorers [9,11,15]. Due to the promising performance, we consider the two SSL-based models [9,15] as strong baselines of this work.…”

“…Automatic scoring of fluency, serves as an essential module in computer-aided language learning (CALL) systems. It has been extensively studied in both "read aloud" [6][7][8][9][10][11] and "open response" [12][13][14][15] scenarios. In the read aloud" scenario, L2 learners are required to read a provided prompt text, whereas the 'open response" requires them to express their opinions freely based on a given question.…”

“…Fluency related features are then extracted and fed into subsequent fluency scorers. Although recent end-to-end neural network based fluency scorers have achieved satisfactory results [7][8][9][10][11]15], their performances heavily rely on the size of labeled scoring samples. In fact, the non-native data labeling process is costly and has scalability issues [16].…”