Subjective Image Quality Assessment With Boosted Triplet Comparisons

Abstract

In subjective full-reference image quality assessment, a reference image is distorted at increasing distortion levels. The differences between perceptual image qualities of the reference image and its distorted versions are evaluated, often using degradation category ratings (DCR). However, the DCR has been criticized since differences between rating categories on this ordinal scale might not be perceptually equidistant, and observers may have different understandings of the categories. Pair comparisons (PC) of distorted images, followed by Thurstonian reconstruction of scale values, overcomes these problems. In addition, PC is more sensitive than DCR, and it can provide scale values in fractional, just noticeable difference (JND) units that express a precise perceptional interpretation. Still, the comparison of images of nearly the same quality can be difficult. We introduce boosting techniques embedded in more general triplet comparisons (TC) that increase the sensitivity even more. Boosting amplifies the artefacts of distorted images, enlarges their visual representation by zooming, increases the visibility of the distortions by a flickering effect, or combines some of the above. Experimental results show the effectiveness of boosted TC for seven types of distortion (color diffusion, jitter, high sharpen, JPEG 2000 compression, lens blur, motion blur, multiplicative noise). For our study, we crowdsourced over 1.7 million responses to triplet questions. We give a detailed analysis of the data in terms of scale reconstructions, accuracy, detection rates, and sensitivity gain. Generally, boosting increases the discriminatory power and allows to reduce the number of subjective ratings without sacrificing the accuracy of the resulting relative image quality values. Our technique paves the way to fine-grained image quality datasets, allowing for more distortion levels, yet with high-quality subjective annotations. We also provide the details for Thurstonian scale reconstruction from TC and our annotated dataset, KonFiG-IQA, containing 10 source images, processed using 7 distortion types at 12 or even 30 levels, uniformly spaced over a span of 3 JND units.

Crowdsourced Quality Assessment of Enhanced Underwater Images - a Pilot Study

“…This was done separately for each of the four scenes, respectively sets of 44 images each. Since in this study, we had adopted pair comparison with a ternary choice, responses of type "not sure" were interpreted as two votes, one for "left" and one for "right", both weighted by 1/2, the same as in [18]. We ranked the enhanced images (for each reference) according to the obtained quality scale values.…”

Perception of video quality at a local spatio-temporal horizon

“…Unfortunately, the limitation is in the size of such matrix, since the number of possible comparisons is growing in 𝑂 (𝑁 2 ), 𝑁 being the number of stimuli: introducing efficiency for a subjective protocol. A lot of previous works have focused on active sampling solutions [3,7,17,18,31,42] and more recently with the work of [4,19,22,26,27,34,41]. Where the target is to select the most informative pairs and minimize experimental effort while maintaining accurate estimations and be robustness to bad annotator behavior (e.g.…”

CID22: Large-Scale Subjective Quality Assessment for High Fidelity Image Compression

“…By amplifying the visibility of distortions, boosted triplet comparison [10] improves the discriminative power of pairwise comparisons (PC). By presenting three stimuli (reference and two distorted images), it assesses fidelity, unlike double stimulus PC protocols which effectively assess only appeal.…”