Toward Bridging the Simulated-to-Real Gap: Benchmarking Super-Resolution on Real Data

Abstract

Capturing ground truth data to benchmark super-resolution (SR) is challenging. Therefore, current quantitative studies are mainly evaluated on simulated data artificially sampled from ground truth images. We argue that such evaluations overestimate the actual performance of SR methods compared to their behavior on real images. Toward bridging this simulated-to-real gap, we introduce the Super-Resolution Erlangen (SupER) database, the first comprehensive laboratory SR database of all-real acquisitions with pixel-wise ground truth. It consists of more than 80k images of 14 scenes combining different facets: CMOS sensor noise, real sampling at four resolution levels, nine scene motion types, two photometric conditions, and lossy video coding at five levels. As such, the database exceeds existing benchmarks by an order of magnitude in quality and quantity. This paper also benchmarks 19 popular single-image and multi-frame algorithms on our data. The benchmark comprises a quantitative study by exploiting ground truth data and qualitative evaluations in a large-scale observer study. We also rigorously investigate agreements between both evaluations from a statistical perspective. One interesting result is that top-performing methods on simulated data may be surpassed by others on real data. Our insights can spur further algorithm development, and the publicy available dataset can foster future evaluations.

Toward Real-World Single Image Super-Resolution: A New Benchmark and a New Model

“…To improve the generalization capability, Zhang et al [60] trained their model using multiple simulated degradations and Bulat et al [6] used a GAN [17] to generate the degradation process. Although these more advanced methods can simulate more complex degradation, there is no guarantee that such simulated degradation can approximate the authentic degradation in practical scenarios which is usually very complicated [26].…”

“…Qu et al [39] put two cameras together with a beam splitter to collect a dataset with paired face images. Köhler et al [26] employed hardware binning on the sensor to capture LR images and used multiple postprocessing steps to generate different versions of an LR image. However, both the datasets were collected in indoor laboratory environment and very limited number of scenes (31 face images in [39] and 14 scenes in [26]) were included.…”

“…Köhler et al [26] employed hardware binning on the sensor to capture LR images and used multiple postprocessing steps to generate different versions of an LR image. However, both the datasets were collected in indoor laboratory environment and very limited number of scenes (31 face images in [39] and 14 scenes in [26]) were included. Different from them, our dataset is constructed by adjusting the focal length of DSLR cameras which naturally results in image pairs at different resolutions, and it contains 234 scenes in both indoor and outdoor environments.…”

“…Though significant advances have been made, most of the existing SISR methods are trained and evaluated on simulated datasets which assume simple and uniform degradation (i.e., bicubic degradation). Unfortunately, SISR models trained on such simulated datasets are hard to generalize to practical applications since the authentic degradations in real-world LR images are much more complex [54,26]. Fig.…”

“…However, constructing such a real-world super-resolution (RealSR) dataset is a non-trivial job since the ground-truth HR images are very difficult to obtain. To the best of our knowledge, only two recent attempts have been made in the laboratory environment, where complicated devices were used to collect image pairs on very limited scenes [39,26]. In this work, we aim to construct a more general and practical RealSR dataset using a flexible and easy-to-reproduce method.…”