Waterloo Exploration Database: New Challenges for Image Quality Assessment Models

Abstract

The great content diversity of real-world digital images poses a grand challenge to image quality assessment (IQA) models, which are traditionally designed and validated on a handful of commonly used IQA databases with very limited content variation. To test the generalization capability and to facilitate the wide usage of IQA techniques in real-world applications, we establish a large-scale database named the Waterloo Exploration Database, which in its current state contains 4744 pristine natural images and 94 880 distorted images created from them. Instead of collecting the mean opinion score for each image via subjective testing, which is extremely difficult if not impossible, we present three alternative test criteria to evaluate the performance of IQA models, namely, the pristine/distorted image discriminability test, the listwise ranking consistency test, and the pairwise preference consistency test (P-test). We compare 20 well-known IQA models using the proposed criteria, which not only provide a stronger test in a more challenging testing environment for existing models, but also demonstrate the additional benefits of using the proposed database. For example, in the P-test, even for the best performing no-reference IQA model, more than 6 million failure cases against the model are "discovered" automatically out of over 1 billion test pairs. Furthermore, we discuss how the new database may be exploited using innovative approaches in the future, to reveal the weaknesses of existing IQA models, to provide insights on how to improve the models, and to shed light on how the next-generation IQA models may be developed. The database and codes are made publicly available at: https://ece.uwaterloo.ca/~k29ma/exploration/.

Blind Image Quality Assessment Using a Deep Bilinear Convolutional Neural Network

“…P-Test measures the pairwise concordance of BIQA models on image pairs with clearly discriminable perceptual quality. More details of the three criteria can be found in [18]. Here we use them to test the robustness of DB-CNN on the Waterloo Exploration Database.…”

“…After that, we compare the performance of DB-CNN with state-of-the-art BIQA models on individual databases and across databases. We also test the robustness of DB-CNN on the Waterloo Exploration Database using the discriminability and ranking consistency criteria [18], and the gMAD competition method. Finally, we conduct a series of ablation experiments to justify the rationality of DB-CNN.…”

“…To take into account the enormous content variations in real-world images, we start with the Waterloo Exploration Database [18] and the PASCAL VOC Database [19]. The former contains 4, 744 pristine-quality images with four synthetic distortions, i.e., JPEG compression, JPEG2000 compression, Gaussian blur, and while Gaussian noise.…”

RVSIM: a feature similarity method for full-reference image quality assessment

“…This study runs the RVSIM index on five image databases, namely, LIVE [45], CSIQ [54], TID2008 [55], TID2013 [56], and Waterloo Exploration database [57], to verify the performance of the proposed method. The five image databases are used here for algorithm validation and comparison.…”

eCNN

“…The scanning uses lightweight settings for speeding up the process, and then the other two apply heavy settings for improving image quality. We train on two datasets, DIV2K [1] and Waterloo Exploration [40], for the following comparison with the state-of-the-art networks of SR and denoising, respectively.…”