Strategy for Boosting Pair Comparison and Improving Quality Assessment Accuracy

Abstract

The development of rigorous quality assessment model relies on the collection of reliable subjective data, where the perceived quality of visual multimedia is rated by the human observers. Different subjective assessment protocols can be used according to the objectives, which determine the discriminability and accuracy of the subjective data. Single stimulus methodology, e.g., the Absolute Category Rating (ACR) has been widely adopted due to its simplicity and efficiency. However, Pair Comparison (PC) is of significant advantages over ACR in terms of discriminability. In addition, PC avoids the influence of observers' bias regarding to their understanding of the quality scale. Nevertheless, full pair comparison is much more time consuming. In this study, we therefore 1) employ a generic model to bridge the pair comparison data and ACR data, where the variance term could be recovered and the obtained information is more complete; 2) propose a fusion strategy to boost pair comparisons by utilizing the ACR results as initialization information; 3) develop a novel active batch sampling strategy based on Minimum Spanning Tree (MST) for PC. In such a way, the proposed methodology could achieve the same accuracy of pair comparison but with the compelxity as low as ACR. Extensive experimental results demonstrate the efficiency and accuracy of the proposed approach, which outperforms the state of the art approaches.

Perception of video quality at a local spatio-temporal horizon

“…Unfortunately, the limitation is in the size of such matrix, since the number of possible comparisons is growing in 𝑂 (𝑁 2 ), 𝑁 being the number of stimuli: introducing efficiency for a subjective protocol. A lot of previous works have focused on active sampling solutions [3,7,17,18,31,42] and more recently with the work of [4,19,22,26,27,34,41]. Where the target is to select the most informative pairs and minimize experimental effort while maintaining accurate estimations and be robustness to bad annotator behavior (e.g.…”

Predictive Sampling for Efficient Pairwise Subjective Image Quality Assessment

“…Some other studies boost PC with data obtained from a single (or double) stimulus subjective test to avoid doing a complete pairwise comparison test, selecting more informative pairs and thus obtaining finer discrimination. In [11] a fusion strategy is introduced to combine ACR and PC subjective tests by initializing PC with ACR subjective test results to achieve higher accuracy. Furthermore, [12] introduces a hybrid approach capable of merging MOS and PC tests through a probabilistic model and active sampling approach.…”