Speech Intelligibility Prediction Using Spectro-Temporal Modulation Analysis

Edraki, Amin; Chan, Wai-Yip; Jensen, Jesper; Fogerty, Daniel

doi:10.1109/taslp.2020.3039929

Cited by 16 publications

(11 citation statements)

References 74 publications

(143 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…STM-based representations of acoustical stimuli have also successfully been applied to computational models of speech representation [ 38 , 39 ]. Furthermore, the ability to detect STM has been shown to be related to speech understanding in listeners with normal pure-tone detection thresholds [ 40 , 41 , 42 ], in listeners with cochlear damage [ 3 , 4 , 43 ], and in listeners who use cochlear implants [ 44 , 45 , 46 ].…”

Section: Introductionmentioning

confidence: 99%

Relating Suprathreshold Auditory Processing Abilities to Speech Understanding in Competition

et al. 2022

View full text Add to dashboard Cite

(1) Background: Difficulty hearing in noise is exacerbated in older adults. Older adults are more likely to have audiometric hearing loss, although some individuals with normal pure-tone audiograms also have difficulty perceiving speech in noise. Additional variables also likely account for speech understanding in noise. It has been suggested that one important class of variables is the ability to process auditory information once it has been detected. Here, we tested a set of these “suprathreshold” auditory processing abilities and related them to performance on a two-part test of speech understanding in competition with and without spatial separation of the target and masking speech. Testing was administered in the Portable Automated Rapid Testing (PART) application developed by our team; PART facilitates psychoacoustic assessments of auditory processing. (2) Methods: Forty-one individuals (average age 51 years), completed assessments of sensitivity to temporal fine structure (TFS) and spectrotemporal modulation (STM) detection via an iPad running the PART application. Statistical models were used to evaluate the strength of associations between performance on the auditory processing tasks and speech understanding in competition. Age and pure-tone-average (PTA) were also included as potential predictors. (3) Results: The model providing the best fit also included age and a measure of diotic frequency modulation (FM) detection but none of the other potential predictors. However, even the best fitting models accounted for 31% or less of the variance, supporting work suggesting that other variables (e.g., cognitive processing abilities) also contribute significantly to speech understanding in noise. (4) Conclusions: The results of the current study do not provide strong support for previous suggestions that suprathreshold processing abilities alone can be used to explain difficulties in speech understanding in competition among older adults. This discrepancy could be due to the speech tests used, the listeners tested, or the suprathreshold tests chosen. Future work with larger numbers of participants is warranted, including a range of cognitive tests and additional assessments of suprathreshold auditory processing abilities.

show abstract

Section: Introductionmentioning

confidence: 99%

Relating Suprathreshold Auditory Processing Abilities to Speech Understanding in Competition

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The other category is derived based on the observation that reverberation and/or additive noise tends to reduce the modulation depth of the distorted signal, compared with the clean reference signal. Well-known approaches of this category include the speech transmission index (STI) [25], spectro-temporal modulation index (STMI), normalized-covariance measure (NCM) [26], short-time objective intelligibility (STOI) [28], extended STOI (eSTOI) [29], polynomial measure (SOPM) [32], and weighted spectro-temporal modulation index (wSTMI) [33]. To avoid the necessity for clean reference speech, several non-intrusive approaches have been proposed.…”

Section: Introductionmentioning

confidence: 99%

Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Features

Zezario¹,

Fu²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

The non-intrusive speech assessment metrics have garnered significant attention in recent years, and several deep learning-based models have been developed accordingly. Although these models are more flexible than conventional speech assessment metrics, most of them are designed to estimate a specific evaluation score, whereas speech assessment generally involves multiple facets. Herein, we propose a cross-domain multiobjective speech assessment model called MOSA-Net, which can estimate multiple speech assessment metrics simultaneously. More specifically, MOSA-Net is designed to estimate the speech quality, intelligibility, and distortion assessment scores of an input test speech signal. It comprises a convolutional neural network and bidirectional long short-term memory (CNN-BLSTM) architecture for representation extraction, and a multiplicative attention layer and a fully-connected layer for each assessment metric. In addition, cross-domain features (spectral and timedomain features) and latent representations from self-supervised learned models are used as inputs to combine rich acoustic information from different speech representations to obtain more accurate assessments. Experimental results show that MOSA-Net can precisely predict perceptual evaluation of speech quality (PESQ), short-time objective intelligibility (STOI), and speech distortion index (SDI) scores when tested on noisy and enhanced speech utterances under either seen test conditions or unseen test conditions. Moreover, MOSA-Net, originally trained to assess objective scores, can be used as a pre-trained model to be effectively adapted to an assessment model for predicting subjective quality and intelligibility scores with a limited amount of training data. In light of the confirmed prediction capability, we further adopt the latent representations of MOSA-Net to guide the speech enhancement (SE) process and derive a quality-intelligibility (QI)aware SE (QIA-SE) approach accordingly. Experimental results show that QIA-SE provides superior enhancement performance compared with the baseline SE system in terms of objective evaluation metrics and qualitative evaluation test.

show abstract

“…This neural network is based on U-net [50], which consists of multiple layers of convolution and transpose convolution at encoder and decoder, respectively. Differently from neural network, wSTMI [51] proposes a linear model, whose weight parameters are optimized through training. wSTMI is based on STMI [52], which predicts speech intelligibility by calculating the correlation coefficient between the spectro-temporal modulation spectrograms of the clean and the degraded signals.…”

Section: Speech Intelligibility Predictionmentioning

confidence: 99%

“…In [41], where SIIB and SIIB Gauss are compared with ESTOI, HASPI, and the other nine SIPs, SIIB and SIIB Gauss perform best among 13 data sets. In [51], the latest developed wSTMI was compared with SIIB, SIIB Gauss , ESTOI, HASPI, and the other nine SIPs. By using some new data sets, wSTMI performs best, with ESTOI ranked second, HASPI ranked third, and SIIB Gauss ranked fourth.…”

Section: Speech Intelligibility Predictionmentioning

confidence: 99%

“…After the articulation index theory, many speech intelligibility predictors have been proposed. These predictors are based on different categories that include the generic SNR [45,47,55,56], the correlation coefficient [44,51,57], neural network [48,49], and mutual information [13,41]. In the following section, we will give an overview of the state-of-the-art speech intelligibility predictors from each category.…”

Section: Speech Intelligibility Predictionmentioning

confidence: 99%

See 1 more Smart Citation

Mutual information based speech intelligibility prediction and its application to hearing aid fitting

Wang¹

View full text Add to dashboard Cite

Speech enhancement was, is, and will be the key technology for digital speech transmission. When developing speech enhancement algorithms, the intelligibility or the quality of the processed speech needs to be evaluated. Intelligibility is more fundamental than quality. The evaluation of speech intelligibility can be carried out through subjective listening tests and objective metrics. Carrying out subjective listening tests is time-consuming and costly. Using the objective metrics is more efficient. Thus, there has been an increasing research in speech intelligibility prediction. Speech intelligibility can be measured in terms of the information received by a listener. This thesis aims at developing a mutual information-based speech intelligibility predictor (SIP) and using the mutual information-based predictor to assist the fitting process in hearing instruments. To achieve this goal, this thesis carried out three studies. First, it studied the modeling of the transmitted message. For mutual information-based SIPs, the most important thing is to determine the transmitted message. This thesis studied two approaches of modeling of the message: one is the continuous-valued sound, and the other is the discrete-valued linguistic message. Two corresponding SIPs were developed. By comparing their predicted intelligibility results with the psychometric curves, which are the subjective intelligibility scores, it shows that the modeling of discrete-valued message gives a better match to the psychometric curves. Second, based on the modeling of the discrete-valued message, this thesis proposed a mutual information-based SIP. Since the discrete-valued message cannot be obtained from a speech signal, the proposed SIP calculates the mutual information between the clean speech and the received speech, instead of calculating the mutual information between the message and the received speech. The proposed SIP considers frequency correlation for both the clean speech and the received speech. The evaluation results show that the proposed SIP performs better than the existing state-of-the-art mutual information-based SIPs. Third, this thesis proposed an automatic fitting tool for the nonlinear frequency compression (NFC) operator, which is a frequency lowering operator used in hearing instruments. The automatic fitting tool adjusts the parameter in the NFC by maximizing the mutual information between the message and the frequency-lowered speech. To evaluate the automatic fitting tool, the parameter was also searched by listening tests. The results show that the parameter determined by the automatic fitting tool is consistent to the parameter determined by the listening tests.

show abstract

Speech Intelligibility Prediction Using Spectro-Temporal Modulation Analysis

Cited by 16 publications

References 74 publications

Relating Suprathreshold Auditory Processing Abilities to Speech Understanding in Competition

Relating Suprathreshold Auditory Processing Abilities to Speech Understanding in Competition

Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Features

Mutual information based speech intelligibility prediction and its application to hearing aid fitting

Contact Info

Product

Resources

About