Using Fluency Representation Learned from Sequential Raw Features for Improving Non-native Fluency Scoring

Abstract

Speech fluency/disfluency can be evaluated by analyzing a range of phonetic and prosodic features. Deep neural networks are commonly trained to map fluency-related features into the human scores. However, the effectiveness of deep learning-based models is constrained by the limited amount of labeled training samples. To address this, we introduce a self-supervised learning (SSL) approach that takes into account phonetic and prosody awareness for fluency scoring. Specifically, we first pre-train the model using a reconstruction loss function, by masking phones and their durations jointly on a large amount of unlabeled speech and text prompts. We then fine-tune the pre-trained model using human-annotated scoring data. Our experimental results, conducted on datasets such as Speechocean762 and our non-native datasets, show that our proposed method outperforms the baseline systems in terms of Pearson correlation coefficients (PCC). Moreover, we also conduct an ablation study to better understand the contribution of phonetic and prosody factors during the pre-training stage.

An ASR-free Fluency Scoring Approach with Self-Supervised Learning

“…To verify the performance of the proposed approach in the "read aloud" scenario, we further conduct experiments on two more datasets, ByteRead and Speechocean762. ByteRead is another internal dataset collected under the "read aloud" scenario and a detailed description is given in [16]. Speechocean762 is an open-sourced speech assessment corpus with 5,000 utterances collected from 250 speakers [27].…”

“…In [16], an ASR-based fluency scoring system was reported with encouraging results in the "read aloud" scenario. To extend the system to the "open response" scenario, an additional step of speech-to-text conversion can be adopted.…”

“…• Deep feature+ASR-based [16]: Figure 1 (a) shows the architecture of this framework. The model inputs are 512-dim deep features extracted from the penultimate layer of the acoustic model (Section 3.2).…”

“…• Proposed SSL+ASR-free: The main network architecture basically follows the ASR-based fluency scoring system in [16], including a preprocessing layer and a two-layer BLSTM. The preprocessing layer consists of a linear layer followed by a Layer-Norm and Tanh activation function.…”

“…comprise additional prosodic and SSL speech features [15] . In [16], Bi-directional Long Short Term Memory (BLSTM) is adopted to predict fluency scores from a sequence of phone-level deep features (usually extracted from the bottleneck layer of a DNN acoustic model) and its corresponding phone duration features.…”