Pre-trained Speech Representations as Feature Extractors for Speech Quality Assessment in Online Conferencing Applications

Abstract

Speech quality in online conferencing applications is typically assessed through human judgements in the form of the mean opinion score (MOS) metric. Since such a labor-intensive approach is not feasible for large-scale speech quality assessments in most settings, the focus has shifted towards automated MOS prediction through end-to-end training of deep neural networks (DNN). Instead of training a network from scratch, we propose to leverage the speech representations from the pre-trained wav2vec-based XLS-R model. However, the number of parameters of such a model exceeds task-specific DNNs by several orders of magnitude, which poses a challenge for resulting fine-tuning procedures on smaller datasets. Therefore, we opt to use pre-trained speech representations from XLS-R in a feature extraction rather than a fine-tuning setting, thereby significantly reducing the number of trainable model parameters. We compare our proposed XLS-R-based feature extractor to a Mel-frequency cepstral coefficient (MFCC)-based one, and experiment with various combinations of bidirectional long short term memory (Bi-LSTM) and attention pooling feedforward (AttPoolFF) networks trained on the output of the feature extractors. We demonstrate the increased performance of pre-trained XLS-R embeddings in terms a reduced root mean squared error (RMSE) on the ConferencingSpeech 2022 MOS prediction task.

On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesis

“…Another recent challenge [24] focused on predicting speech quality in speech conferencing applications, and also saw several submissions, e.g. [25,26], making use of SSL representations. This task does involve spontaneous speech audio, but focuses only on assessing quality of speech transmission in online conferencing and not on asessing synthesized spontaneous speech from a TTS model.…”

Non-intrusive Speech Intelligibility Metric Prediction for Hearing Impaired Individuals

“…SSSRs have been applied to metric prediction tasks, typically to quality prediction [22,23]. In [13], XLSR representations are used as feature extraction in a non-intrusive human MOS prediction network. Similarly, in [24] SSSRs are used for the same quality prediction task, but they are fine-tuned with a mean pooling layer rather than being used simply as feature extraction.…”

“…A model structure inspired by [13] is chosen for the SI prediction network. Five feature extraction methods are used; outputs of G FE and G OL for both, HuBERT and XLSR representations, as well as a spectrogram representation denoted as SPEC.…”

“…It is understood that SSSRs are able to encode and predict the context of the speech content in the input audio, and thus model the patterns of spoken language. Recent work [12][13][14][15][16] has found that in addition to speech content, SSSRs are also able to capture information on potentially corrupting noise and distortion in the input audio. In this work, SSSRs are used as a feature transformation for non-intrusive neural speech intelligibility prediction networks, trained on the CPC1 challenge dataset.…”

Perceive and Predict: Self-Supervised Speech Representation Based Loss Functions for Speech Enhancement

“…It is demonstrated the encoder layers have a notably stronger correlation to the aforementioned evaluation measures than the output layers. Hidden unit BERT (HuBERT) [12] and XLSR [14] SSSRs are chosen as these have both been applied in related speech tasks previously but in different ways [15,16]. Following from this, the distances between clean and noisy SSSR features are then evaluated for their usefulness as loss functions to train speech enhancement models.…”