Flexible Style Image Super-Resolution Using Conditional Objective

Abstract

Recent studies have significantly enhanced the performance of single-image super-resolution (SR) using convolutional neural networks (CNNs). While there can be many high-resolution (HR) solutions for a given input, most existing CNN-based methods do not explore alternative solutions during the inference. A typical approach to obtaining alternative SR results is to train multiple SR models with different loss weightings and exploit the combination of these models. Instead of using multiple models, we present a more efficient method to train a single adjustable SR model on various combinations of losses by taking advantage of multi-task learning. Specifically, we optimize an SR model with a conditional objective during training, where the objective is a weighted sum of multiple perceptual losses at different feature levels. The weights vary according to given conditions, and the set of weights is defined as a style controller. Also, we present an architecture appropriate for this training scheme, which is the Residual-in-Residual Dense Block equipped with spatial feature transformation layers. At the inference phase, our trained model can generate locally different outputs conditioned on the style control map. Extensive experiments show that the proposed SR model produces various desirable reconstructions without artifacts and yields comparable quantitative performance to state-of-the-art SR methods. Code and trained models are available at https://github.com/seungho-snu/FxSR.

Two-Stage Domain Adapted Training For Better Generalization In Real-World Image Restoration And Super-Resolution

“…Similar conclusions to the quantitative comparisons can be drawn from qualitative comparisons. We observe that all GAN-SR results including ESRGAN-FS [12], ESR-GAN+ [50], SPSR [40], RankSRGAN [67], LDL [31] and FxSR [47] produce visible artifacts and experience excessive sharpness. On the other hand, our method WGSR is able to reconstruct the genuine image details with high reconstruction accuracy including the regions with regular patterns and the areas containing fine details such as the light pink flowers on the bush (Fig.…”

“…Liang et al [31] suggested a locally discriminative learning framework LDL by externally computing a probability map of each pixel being artifacts based on patch-level residual variances. Park et al [47] introduced Flexible Style Image Super-Resolution (FxSR), which optimizes SR network with image-specific objectives without considering the regional characteristics. Later, in SROOE, Park et al [48] proposed optimal objective estimation depending on perceptual and objective image maps.…”

“…Later, in SROOE, Park et al [48] proposed optimal objective estimation depending on perceptual and objective image maps. These methods [31,47,48] coexist with the computational burden of a large number of image maps. On the contrary, our wavelet-loss guided model does not require explicit calculation of an artifact map and inherently learns to suppress artifacts while retaining genuine details.…”

“…Table 1 demonstrates quantitative comparison for ×4 SR methods and our proposed approach WGSR. We compare our method with the existing state-of-the-art methods including ESRGAN-FS [12], ESRGAN+ [50], SPSR [40], RankSRGAN [67], SRFlow-DA (heat=0.9) [23], LDL [31], FxSR (t=0.8) [47], PDASR [71] and SROOE (t=0.9) [48]. Our method WGSR improves the perceptual quality and the reconstruction accuracy simultaneously.…”

“…RankSRGAN [67] (25.68 / 0.190) LDL [31] (26.67 / 0.194) PDASR [71] (26.86 / 0.196) FxSR [47] (26.25 / 0.187) WGSR (Ours) (26.69 / 0.193) DualFormer [38] (26.98 / 0.188) SROOE [48] (26.67 / 0.184) HR (PSNR↑ / DISTS↓) Figure 2. Visual performance of recent ×4 SR methods on a crop from Urban100 dataset (img-6) [20].…”