Learned Image Downscaling for Upscaling Using Content Adaptive Resampler

Abstract

Deep convolutional neural network based image super-resolution (SR) models have shown superior performance in recovering the underlying high resolution (HR) images from low resolution (LR) images obtained from the predefined downscaling methods. In this paper we propose a learned image downscaling method based on content adaptive resampler (CAR) with consideration on the upscaling process. The proposed resampler network generates content adaptive image resampling kernels that are applied to the original HR input to generate pixels on the downscaled image. Moreover, a differentiable upscaling (SR) module is employed to upscale the LR result into its underlying HR counterpart. By back-propagating the reconstruction error down to the original HR input across the entire framework to adjust model parameters, the proposed framework achieves a new state-of-the-art SR performance through upscaling guided image resamplers which adaptively preserve detailed information that is essential to the upscaling. Experimental results indicate that the quality of the generated LR image is comparable to that of the traditional interpolation based method, but the significant SR performance gain is achieved by deep SR models trained jointly with the CAR model. The code is publicly available on: URL https://github.com/sunwj/CAR.

How important are faces for person re-identification?

“…Image super resolution techniques (e.g. [45]) are effective for generating realistic high-resolution content from lowresolution imagery. These techniques, however, effectively only hallucinate a plausible realistic high-resolution version of the low-resolution contents, and therefore cannot be used to reveal the true anonymized face.…”

Noise Reconstruction & Removal Network: A New Architecture to Denoise FIB-SEM Images

“…In recent years, deep learning methods have established themselves as powerful analytical tools in machine learning (Arik, Chrzanowski, Coates, Diamos, Gibiansky, Kang, Li, Miller, Ng, Raiman et al 2017, Jing, Yang, Feng, Ye, Yu & Song 2019, Karpathy & Fei-Fei 2015, Oord, Dieleman, Zen, Simonyan, Vinyals, Graves, Kalchbrenner, Senior & Kavukcuoglu 2016. Specifically, Convolutional Neural Networks (CNNs) have been used for various tasks including detection (Lin, Goyal, Girshick, He & Dollár 2018, Liu, Anguelov, Erhan, Szegedy, Reed, Fu & Berg 2016, Redmon & Farhadi 2018, Ren, He, Girshick & Sun 2016, Tan, Pang & Le 2020, segmentation (Chen, Papandreou, Schroff & Adam 2017, Chen, Zhu, Papandreou, Schroff & Adam 2018, Ronneberger, Fischer & Brox 2015, Tao, Sapra & Catanzaro 2020, Yuan, Chen & Wang 2019, super resolution (Sun & Chen 2020), etc. In the field of denoising, CNNs have been very useful (Kim, Chung & Jung 2019, Liu, Wu, Wang, Xu, Zhou, Huang, Wang, Cai, Ding, Fan & Wang 2019, Yu, Park & Jeong 2019 especially when the noise characteristics are unknown, making any mathematical modeling difficult.…”

Satellite Image Multi-Frame Super Resolution Using 3D Wide-Activation Neural Networks

“…In recent years, as deep learning applications explode in every computer-vision task, convolutional neural network methods began to dominate the problem of SR. However, most of them focus on Single-Image Super Resolution (SISR) [14][15][16] and do not take advantage of the temporal information inherent to multi-frame tasks. MFSR has been studied in video; for example, Sajjadi et al [17] proposed a framework that uses the previously inferred HR estimate to super resolve the subsequent frame; Jo et al [18] created an end-to-end deep neural network that generates dynamic upsampling filters and a residual image avoiding explicit motion compensation; and Kim et al [19] presented 3DSRnet, a framework that maintains the temporal depth of spatio-temporal feature maps to capture nonlinear characteristics between low-and high-resolution frames.…”