Perceptually based downscaling of images

Abstract

We propose a perceptually based method for downscaling images that provides a better apparent depiction of the input image. We formulate image downscaling as an optimization problem where the difference between the input and output images is measured using a widely adopted perceptual image quality metric. The downscaled images retain perceptually important features and details, resulting in an accurate and spatio-temporally consistent representation of the high resolution input. We derive the solution of the optimization problem in closed-form, which leads to a simple, efficient and parallelizable implementation with sums and convolutions. The algorithm has running times similar to linear filtering and is orders of magnitude faster than the state-of-the-art for image downscaling. We validate the effectiveness of the technique with extensive tests on many images, video, and by performing a user study, which indicates a clear preference for the results of the new algorithm.

Learned Image Downscaling for Upscaling Using Content Adaptive Resampler

“…The results indicate that the CAR model achieves at least 75% preference over all other algorithms. Besides, our algorithm achieves more than 98% preference compared with the Perceptually method, demonstrating that there is a distinct difference between the SR image corresponding to the perceptually based [15] downscaled image and the original HR image. Although there are about 25% and 20% preference on 'A equals to B' plus 'B is better than A' on the DPID and L0-regularized entry, respectively, it still cannot compete with the significant superiority of the CAR method.…”

“…This section reports the quantitative and qualitative performance of different image downscaling methods for SR. Then ablation studies of the proposed CAR model is conducted. We compare the CAR model with four baseline methods, i.e., the bicubic downscaling (Bicubic), and other three state-of-the-art image downscaling methods: perceptually optimized image downscaling (Perceptually) [15], detail-preserving image downscaling (DPID) [17], and L0-regularized image downscaling (L0-regularized) [19]. We train SR models using LR images downscaled by those four baseline downscaling algorithms and LR images downscaled by the proposed CAR model.…”

“…The key idea is to optimize the shape and locations of the downsampling kernels to better align with local image features by considering both spatial and color variances of the local region. Öztireli and Gross [15] proposed a method to downscale HR images without filtering. They consider image downscaling as an optimization problem and use the structural similarity index (SSIM) [16] as objective to directly optimize the downscaled image against its original image, and this approach helps to capture most of the perceptually important details.…”

“…We adopt similar evaluation settings used in [14,15,17,19]. The user study are conducted as the A/B testing: the original image is presented in the middle place with two variants (SR images or downscaled images) showed in either side, among which one is produced by the CAR method and another is generated by one of the competing methods, i.e., Bicubic, Perceptually [15], DPID [17] and L0-regularized [19]. The users are required to ask the question 'which one looks better' by exclusively selecting one of the three options from: 1) A is better than B; 2) A equals to B; 3) B is better than A.…”

“…We invited 34 participants for both the user study on 4× image super-resolution and 4× image downscaling tasks. Answers from each participant are filtered out if it achieves less than 80% consistency [14,15] on the repeated 10 questions, which generates 33 and 30 valid records for the image super-resolution and image downscaling tasks, respectively. As can be seen from Fig.…”