Evaluating instrumental measures of speech quality using Bayesian model selection: Correlations can be misleading!

Kolossa, Antonio; Abel, Johannes; Fingscheidt, Tim

doi:10.1109/icassp.2016.7471752

Cited by 2 publications

(3 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…BMS methods are widely applied in many fields (Raftery, 1995; Hoeting et al, 1999; Penny and Roberts, 2002; Pitt and Myung, 2002; Beal and Ghahramani, 2003; Kemp et al, 2007; Hoijtink et al, 2008; Vyshemirsky and Girolami, 2008; Toni et al, 2009; Penny et al, 2010; Kolossa et al, 2016). We used a two-level hierarchical general linear model (GLM) with the Parametric Empirical Bayesian (PEB) scheme and random effects BMS for group studies as implemented in the SPM software (Friston et al, 2002, 2007; Stephan et al, 2009).…”

Section: Methodsmentioning

confidence: 99%

Mind the Noise When Identifying Computational Models of Cognition from Brain Activity

Kolossa

Kopp

2016

Front. Neurosci.

Self Cite

View full text Add to dashboard Cite

The aim of this study was to analyze how measurement error affects the validity of modeling studies in computational neuroscience. A synthetic validity test was created using simulated P300 event-related potentials as an example. The model space comprised four computational models of single-trial P300 amplitude fluctuations which differed in terms of complexity and dependency. The single-trial fluctuation of simulated P300 amplitudes was computed on the basis of one of the models, at various levels of measurement error and at various numbers of data points. Bayesian model selection was performed based on exceedance probabilities. At very low numbers of data points, the least complex model generally outperformed the data-generating model. Invalid model identification also occurred at low levels of data quality and under low numbers of data points if the winning model's predictors were closely correlated with the predictors from the data-generating model. Given sufficient data quality and numbers of data points, the data-generating model could be correctly identified, even against models which were very similar to the data-generating model. Thus, a number of variables affects the validity of computational modeling studies, and data quality and numbers of data points are among the main factors relevant to the issue. Further, the nature of the model space (i.e., model complexity, model dependency) should not be neglected. This study provided quantitative results which show the importance of ensuring the validity of computational modeling via adequately prepared studies. The accomplishment of synthetic validity tests is recommended for future applications. Beyond that, we propose to render the demonstration of sufficient validity via adequate simulations mandatory to computational modeling studies.

show abstract

Section: Methodsmentioning

confidence: 99%

Mind the Noise When Identifying Computational Models of Cognition from Brain Activity

Kolossa

Kopp

2016

Front. Neurosci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Please note that selecting the best model based on the LCC performance may be misleading, especially when the values of the compared LCCs are small or very close, as shown in [54]: Kolossa et al proposed a sophisticated Bayesian model selection (BMS) method showing by example that a model with an LCC of 0.89 can be statistically significantly better than a model with an LCC of 0.90. In another case, the model with an LCC of 0.31 is -by BMS -only very weakly preferred to the one with a much lower LCC of 0.24, considering statistical evidence.…”

Section: Performance Metricsmentioning

confidence: 99%

“…However, the models selected by statistical evidence (BMS) always offer lower distance values measured between the predictions and their corresponding ground truth. Therefore, instead of employing the sophisticated BMS methods in our case, we decided to report the 95% confidence interval for the MAE measurement, but refrained from further analysis of variance or confidence intervals with the LCC -for the nontrivial reasons and observations in [54]. We entered 95% confidence intervals for MAE wherever space allowed into Tabs.…”

Section: Performance Metricsmentioning

confidence: 99%

Coded Speech Quality Measurement by a Non-Intrusive PESQ-DNN

Xu,

Zhao,

Fingscheidt

2023

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

Wideband codecs such as AMR-WB or EVS are widely used in (mobile) speech communication. Evaluation of coded speech quality is often performed subjectively by an absolute category rating (ACR) listening test. However, the ACR test is impractical for online monitoring of speech communication networks. Perceptual evaluation of speech quality (PESQ) is one of the widely used metrics instrumentally predicting the results of an ACR test. However, the PESQ algorithm requires an original reference signal, which is usually unavailable in network monitoring, thus limiting its applicability. NISQA is a new nonintrusive neural-network-based speech quality measure, focusing on super-wideband speech signals. In this work, however, we aim at predicting the well-known PESQ metric using a non-intrusive PESQ-DNN model. We illustrate the potential of this model by predicting the PESQ scores of wideband-coded speech obtained from AMR-WB or EVS codecs operating at different bitrates in noisy, tandeming, and error-prone transmission conditions. We compare our methods with the state-of-the-art network topologies of QualityNet, WaweNet, and DNSMOS-all applied to PESQ prediction-by measuring the mean absolute error (MAE) and the linear correlation coefficient (LCC). The proposed PESQ-DNN offers the best total MAE and LCC of 0.11 and 0.92, respectively, in conditions without frame loss, and still is best when including frame loss. Note that our model could be similarly used to non-intrusively predict POLQA or other (intrusive) metrics. The proposed PESQ-DNN model definition and the code are provided at https://github.com/ifnspaml/PESQDNN.

show abstract

Evaluating instrumental measures of speech quality using Bayesian model selection: Correlations can be misleading!

Cited by 2 publications

References 21 publications

Mind the Noise When Identifying Computational Models of Cognition from Brain Activity

Mind the Noise When Identifying Computational Models of Cognition from Brain Activity

Coded Speech Quality Measurement by a Non-Intrusive PESQ-DNN

Contact Info

Product

Resources

About