Feature Representation Matters: End-to-End Learning for Reference-Based Image Super-Resolution

Xie, Yanchun; Xiao, Jimin; Sun, Mingjie; Yao, Chao; Huang, Kaizhu

doi:10.1007/978-3-030-58548-8_14

Cited by 37 publications

(39 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3) Reference-based SR: Reference-based SR (RefSR) aims at super-resolving an LR image using one (e.g., [41]- [44]) or multiple (e.g., [45], [46]) external HR images as reference. Yue et al [45] proposed a RefSR method by retrieving similar reference images from web.…”

Section: B Multi-image Srmentioning

confidence: 99%

“…More recently, Yan et al [46] proposed a contentindependent multi-reference SR method to adaptively match the visual pattern between reference and target images. Xie et al [44] improved SRNTT [42] by using a task-specific feature extractor for feature matching and swapping and designed an end-to-end training framework for RefSR. 4) Stereo image SR: The disparities in stereo images are much larger than those in video frames and LF images, and the resolution of both left and right images is low.…”

Section: B Multi-image Srmentioning

confidence: 99%

See 1 more Smart Citation

Learning Parallax Attention for Stereo Image Super-Resolution

Wang

Liang³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

245

228

View full text Add to dashboard Cite

Stereo image pairs can be used to improve the performance of super-resolution (SR) since additional information is provided from a second viewpoint. However, it is challenging to incorporate this information for SR since disparities between stereo images vary significantly. In this paper, we propose a parallax-attention stereo superresolution network (PASSRnet) to integrate the information from a stereo image pair for SR. Specifically, we introduce a parallax-attention mechanism with a global receptive field along the epipolar line to handle different stereo images with large disparity variations. We also propose a new and the largest dataset for stereo image SR (namely, Flickr1024). Extensive experiments demonstrate that the parallax-attention mechanism can capture correspondence between stereo images to improve SR performance with a small computational and memory cost. Comparative results show that our PASSRnet achieves the state-of-the-art performance on the Middlebury, KITTI 2012 and KITTI 2015 datasets.

show abstract

Section: B Multi-image Srmentioning

confidence: 99%

Section: B Multi-image Srmentioning

confidence: 99%

Learning Parallax Attention for Stereo Image Super-Resolution

Wang

Liang³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

245

228

View full text Add to dashboard Cite

show abstract

“…With a fixed VGG network as feature extractors, their method does not train the extractor jointly with the reconstruction net. Yang et al [35] and Xie et al [34] further proposed to adopt a learnable extractor and replace Patch Match with a patch-based attention, which allows an end-to-end learning pipeline. These patch-level warping methods can find semantic-similar patches, but are non-robust to inter-patch misalignment (e.g.…”

Section: Reference-based Super Resolutionmentioning

confidence: 99%

“…It has been widely observed that similar semantic patches and texture patterns tend to recur in the same or highly-correlated images with variable positions, orientations and sizes [23,46]. To search and utilize these correlated patterns from reference images, previous learning-based approaches adopt either patch-wise matching (patch-match [44,43], patchbased attention [34,35]) or pixel-wise alignment (opticalflow [45], offsets [28]), with different pros and cons. The pixel-wise alignment is able to handle non-rigid transformation, but usually less stable and prone to generate distorted structures due to the difficulty of reliable flow or offsets estimations [6], especially for largely misaligned reference images.…”

Section: Introductionmentioning

confidence: 99%

Dual-Camera Super-Resolution with Aligned Attention Modules

Wang

Xie²,

Sun³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present a novel approach to reference-based superresolution (RefSR) with the focus on dual-camera superresolution (DCSR), which utilizes reference images for high-quality and high-fidelity results. Our proposed method generalizes the standard patch-based feature matching with spatial alignment operations. We further explore the dualcamera super-resolution that is one promising application of RefSR, and build a dataset that consists of 146 image pairs from the main and telephoto cameras in a smartphone. To bridge the domain gaps between realworld images and the training images, we propose a selfsupervised domain adaptation strategy for real-world images. Extensive experiments on our dataset and a public benchmark demonstrate clear improvement achieved by our method over state of the art in both quantitative evaluation and visual comparisons. Our code and data are available at https://tengfei-wang.github.io/ Dual-Camera-SR/index.html.

show abstract

“…Zhang et al [43] proposed to search for the matching patches from the reference image in the feature space and then swap the matched features to represent the LR image. Xie et al [44] improved this framework by enhancing the feature extractor. Yang et al [45] applied the attention mechanism to transfer and fuse HR features from the reference image into LR features based on their relevance embedding.…”

Section: B Reference-based Image Super-resolutionmentioning

confidence: 99%

Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses

Jin

Hou

Chen

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple lowresolution cameras. The performance of existing methods is still limited, as they produce either blurry results on plain textured areas or distortions around depth discontinuous boundaries. To tackle this challenge, we propose a novel end-to-end learningbased approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation, while the other module warps another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations adaptively via the learned attention maps, leading to the final high-resolution LF image with satisfactory results on both plain textured areas and depth discontinuous boundaries. Besides, to promote the effectiveness of our method trained with simulated hybrid data on real hybrid data captured by a hybrid LF imaging system, we carefully design the network architecture and the training strategy. Extensive experiments on both real and simulated hybrid data demonstrate the significant superiority of our approach over state-of-the-art ones. To the best of our knowledge, this is the first end-to-end deep learning method for LF reconstruction from a real hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission. The code will be publicly available at https://github.com/jingjin25/LFhybridSR-Fusion.

show abstract

Feature Representation Matters: End-to-End Learning for Reference-Based Image Super-Resolution

Cited by 37 publications

References 25 publications

Learning Parallax Attention for Stereo Image Super-Resolution

Learning Parallax Attention for Stereo Image Super-Resolution

Dual-Camera Super-Resolution with Aligned Attention Modules

Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses

Contact Info

Product

Resources

About