Robust Reference-Based Super-Resolution With Similarity-Aware Deformable Convolution

Shim, Gyumin; Park, Jinsun; Kweon, In So

doi:10.1109/cvpr42600.2020.00845

Cited by 90 publications

(78 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most recently, Yang et al [16] introduced a more accurate way to search and transfer relevant textures from Ref to LSR images. SSEN [17] aligned the Ref and LSR images in the feature domain to capture similarity-aware. In general, these deep methods achieve better results than SISR methods.…”

Section: Reference-based Image Super-resolutionmentioning

confidence: 99%

Unsupervised Remoting Sensing Super-Resolution via Migration Image Prior

Wang

Shao

Lü

et al. 2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

Recently, satellites with high temporal resolution have fostered wide attention in various practical applications. Due to limitations of bandwidth and hardware cost, however, the spatial resolution of such satellites is considerably low, largely limiting their potentials in scenarios that require spatially explicit information. To improve image resolution, numerous approaches based on training low-high resolution pairs have been proposed to address the super-resolution (SR) task. Despite their success, however, low/high spatial resolution pairs are usually difficult to obtain in satellites with a high temporal resolution, making such approaches in SR impractical to use. In this paper, we proposed a new unsupervised learning framework, called "MIP", which achieves SR tasks without low/high resolution image pairs. First, random noise maps are fed into a designed generative adversarial network (GAN) for reconstruction. Then, the proposed method converts the reference image to latent space as the migration image prior. Finally, we update the input noise via an implicit method, and further transfer the texture and structured information from the reference image. Extensive experimental results on the Draper dataset show that MIP achieves significant improvements over state-of-the-art methods both quantitatively and qualitatively. The proposed MIP is open-sourced at https://github.com/jiaming-wang/MIP.

show abstract

Section: Reference-based Image Super-resolutionmentioning

confidence: 99%

Unsupervised Remoting Sensing Super-Resolution via Migration Image Prior

Wang

Shao

Lü

et al. 2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

show abstract

“…For providing useful HF detail to network directly, many referencebased image SR methods [32,44,57,63] attempt to find the small patch recurring in the within and across scale in the reference image. Video super-resolution has more related HF details between frames than image super-resolution due to the extra time dimension.…”

Section: Space Super-resolutionmentioning

confidence: 99%

“…Deformable convolution predicts offsets of the kernel to make a deformed sampling grid possible. We use it in the same way as in feature alignment [44] and time alignment [48,49]. In our work, deformable convolution devotes filling feature holes where the 3D warping performs poorly.…”

Section: Feature Refinement Networkmentioning

confidence: 99%

“…where 𝑦 and 𝑥 are the output and input, and 𝐾, 𝑘, and 𝑤 denote the total number of kernel weights, index and kernel weights. Δ𝑚 𝑘 denotes modulation scalar lying in [0, 1], and it enables relevancyaware weight learning to robustly extract useful correspondences in the reference image, as describe in [44]. 𝑝, 𝑝 𝑘 , and △𝑝 𝑘 denote the center location, the 𝑘-th fixed offset, and the learnable offsets.…”

Section: Feature Refinement Networkmentioning

confidence: 99%

“…Hence we introduce a feature refine network, which utilizes a deformable convolution [8,65] to remove these artifacts. Deformable convolution has been proved to be effective in similarity features searching in the reference image [44] and temporal alignment [48,49] by enlarging the receptive field. It can provide more accurate correspondence between different views in the depth-ambiguous region.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Space-Angle Super-Resolution for Multi-View Images

Sun

Cheng

Yan

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

The limited spatial and angular resolutions in multi-view multimedia applications restrict their visual experience in practical use. In this paper, we first argue the space-angle super-resolution (SASR) problem for irregular arranged multi-view images. It aims to increase the spatial resolution of source views and synthesize arbitrary virtual high resolution (HR) views between them jointly. One feasible solution is to perform super-resolution (SR) and view synthesis (VS) methods separately. However, it cannot fully exploit the intra-relationship between SR and VS tasks. Intuitively, multi-view images can provide more angular references, and higher resolution can provide more high-frequency details. Therefore, we propose a one-stage space-angle super-resolution network called SASRnet, which simultaneously synthesizes real and virtual HR views. Extensive experiments on several benchmarks demonstrate that our proposed method outperforms two-stage methods, meanwhile prove that SR and VS can promote each other. To our knowledge, this work is the first to address the SASR problem for unstructured multi-view images in an end-to-end learning-based manner. CCS CONCEPTS• Computing methodologies → Image-based rendering; Reconstruction; Image processing.

show abstract

Semi‐supervised super‐resolution of diffusion‐weighted images based on multiple references

Guo

Wang

et al. 2023

NMR in Biomedicine

View full text Add to dashboard Cite

Spatial resolution of diffusion tensor images is usually compromised to accelerate the acquisitions, and the state-of-the-art (SOTA) image super-resolution (SR) reconstruction methods are commonly based on supervised learning models.Considering that matched low-resolution (LR) and high-resolution (HR) diffusionweighted (DW) image pairs are not readily available, we propose a semi-supervised DW image SR reconstruction method based on multiple references (MRSR) extracted from other subjects. In MRSR, the prior information of multiple HR reference images is migrated into a residual-like network to assist SR reconstruction of DW images, and a CycleGAN-based semi-supervised strategy is used to train the network with 30% matched and 70% unmatched LR-HR image pairs. We evaluate the performance of the MRSR by comparing against SOTA methods on an HCP dataset in terms of the quality of reconstructed DW images and diffusion metrics. MRSR achieves the best performance, with the mean PSNR/SSIM of DW images being improved by at least 14.3%/28.8% and 1%/1.4% respectively relative to SOTA unsupervised and supervised learning methods, and with the fiber orientations deviating from the ground truth by about 6.28 on average, the RMSEs of fractional anisotropy, mean diffusivity, axial diffusivity and radial diffusivity being 3.0%, 4.6%, 5.7% and 4.5% respectively relative to the ground truth. We validate the effectiveness of the proposed network structure, multiple-reference and CycleGAN-based semi-supervised learning strategies for SR reconstruction of diffusion tensor images through the ablation studies. The proposed method allows us to achieve SR reconstruction for diffusion tensor images with a limited number of matched image pairs.

show abstract

Robust Reference-Based Super-Resolution With Similarity-Aware Deformable Convolution

Cited by 90 publications

References 25 publications

Unsupervised Remoting Sensing Super-Resolution via Migration Image Prior

Unsupervised Remoting Sensing Super-Resolution via Migration Image Prior

Space-Angle Super-Resolution for Multi-View Images

Semi‐supervised super‐resolution of diffusion‐weighted images based on multiple references

Contact Info

Product

Resources

About