An extensive performance evaluation of full-reference HDR image quality metrics

Abstract

High dynamic range (HDR) image and video technology has recently attracted a great deal of attention in the multimedia community, as a mean to produce truly realistic video and further improve the quality of experience (QoE) of emerging multimedia services. In this context, measuring the quality of compressed HDR content plays a fundamental role. However, full-reference (FR) HDR visual quality assessment poses new challenges with respect to the conventional low dynamic range case. Quality metrics have to be redesigned or adapted to HDR, and understanding their reliability to predict users' judgments is even more critical due to the still limited availability of HDR displays to perform subjective evaluations. The goal of this paper is to provide a complete and thorough survey of the performance of the most popular HDR FR image quality metrics. To this end, we gather several existing HDR image databases with subjective quality annotations, in addition to a new one created by ourselves. After aligning the scores in these databases, we obtain an extensive set of 690 compressed HDR images, along with their subjective quality. Next, we analyze in depth many FR metrics, including those used in MPEG standardization, using both classical correlation analyses and classification accuracy. We believe that our results could serve as the most complete and comprehensive benchmark of image quality metrics in the field of HDR image compression.

From Pairwise Comparisons and Rating to a Unified Quality Scale

“…While this aggregation of subjective quality scores is usually done for rating (i.e. mean opinion scores) [1], [2], [19] or pairwise comparisons [20], [21] individually, little has been done to study the fusion of scores obtained by both these two methodologies. In this regard, Ye and Doermann [17] proposed a unified probabilistic model, aggregating rating and pairwise comparisons together.…”

“…For example, an image rated 4 on a 5-point scale in one experiment could be rated 2 in another experiment because of differences in the training, range and type of considered distortions. Dealing with widely different scales when training quality metrics is problematic, often requires using rank-order correlation as a measure of prediction accuracy, and makes difficult the use of multiple datasets for training [1], [2]. M. Pérez-Ortiz and A.…”

Quality Metric Aggregation for HDR/WCG Images

“…In addition those databases are composed of a rather small amount of images. To obtain a suitable database for our experiment, we considered the five databases presented in Table 1: Narwaria et al [21], Korshunov et al [22], Zerman et al [23], 4Kdtb [11] and HDdtb [18]. The first four databases were used for the training phase and HDdtb was considered as an independent test database used to validate our proposed metric.…”

Impacts of Viewing Conditions on HDR-VDP2

“…1) Content creation: Eight images were selected from 3 collections: two are from the MPEG HDR sequences (FireEater and Market) [14], one is from the Stuttgart HDR Video Database [3] and the remaining five images comes from HDR photographic survey [2]. Note that these images also belong to Zerman et al's database [24]. All these images have been encapsulated in the WCG gamut BT.2020 [9] instead of the standard gamut BT.709 [8].…”

“…In [6], authors came to the conclusion that HDR-VDP2 (but in an earlier version 2.1.1) can be successfully used for predicting the quality of video pair comparison contrary to HDR-VQM. More recently, Zerman et al [24], first, combined several existing image databases and, second, they found out that HDR-VQM is the best full-reference HDR quality metric, closely followed by the HDR-VDP2.2.1 metric which gives similar results when one particular database is discarded.…”