On the accuracy of objective image and video quality models: New methodology for performance evaluation

Abstract

International audienceThere are several standard methods for evaluating the performance of models for objective quality assessment with respect to results of subjective tests. However, all of them suffer from one or more of the following drawbacks: They do not consider the uncertainty in the subjective scores, requiring the models to make certain decision where the correct behavior is not known. They are vulnerable to the quality range of the stimuli in the experiments. In order to compare the models, they require a mapping of predicted values to the subjective scores, thus not comparing the models exactly as they are used in the real scenarios. In this paper, new methodology for objective models performance evaluation is proposed. The method is based on determining the classification abilities of the models considering two scenarios inspired by the real applications. It does not suffer from the previously stated drawbacks and enables to easily evaluate the performance on the data from multiple subjective experiments. Moreover, techniques to determine statistical significance of the performance differences are suggested. The proposed framework is tested on several selected metrics and datasets, showing the ability to provide a complementary information about the models' behavior while being in parallel with other state-of-the-art methods

An extensive performance evaluation of full-reference HDR image quality metrics

“…All but two of the considered metrics are Fig. 6 Statistical analysis results for the discriminability analysis, according to the procedure described in Krasula et al (2016). The bars signify statistical equivalence between the quality metrics if they have the same bar aligned with two quality metrics.…”

“…Finally, the results of this approach are generally evaluated in a qualitative manner, e.g., by considering how the number of correct decisions, false rankings, false differentiations, etc., vary as a function of objective metric differences (Brill et al 2004;Hanhart et al 2015b); conversely, a compact, quantitative measure is desirable in order to fairly compare different metrics. Another approach to this problem has been recently proposed by Krasula et al (2016). In their paper, Krasula et al find the accuracy of an objective image or video quality metric by transforming the problem into a classification problem.…”

“…For this purpose, they find z-score of subjective scores and the difference of objective scores for each pair of stimuli, and then find the accuracy of the metric by calculating classification rates. Due to the factors above limiting the effectiveness of resolving power, in this work we propose an alternative approach in the original scale of the metric similar to what has been presented in Krasula et al (2016), which enables to evaluate its discrimination power while avoiding the shortcomings discussed above. Despite the similarities, the Bold value signifies the best result within the same column.…”

“…Given a dataset of subjective scores, it is possible to apply a pairwise statistical test (e.g., a two-way t-test or z-test) to determine whether two MOSs are the same, at a given significance level. In our work, we employ a one-way analysis of variance (ANOVA), with Tukey's honestly significant difference criterion to account for the multiple comparison bias (Hogg and Ledolter 1987), as it is also stated as the ideal way to find statistical significance in Krasula et al (2016). Figure 5a shows the results of ANOVA on our combined database, thresholded at a confidence level of 95% (i.e., 5% significance …”

“…We report for comparison the fraction of correct decisions (CD) at 95% confidence level as proposed in Brill et al (2004) which is to be compared with Acc à . Furthermore, we present the results of statistical significance evaluation of the reported AUC values according to the guidelines presented in Krasula et al (2016). The results of this statistical significance evaluation are presented in Fig.…”