Stereoscopic Image Super-Resolution with Stereo Consistent Feature

Abstract

We present a first attempt for stereoscopic image super-resolution (SR) for recovering high-resolution details while preserving stereo-consistency between stereoscopic image pair. The most challenging issue in the stereoscopic SR is that the texture details should be consistent for corresponding pixels in stereoscopic SR image pair. However, existing stereo SR methods cannot maintain the stereo-consistency, thus causing 3D fatigue to the viewers. To address this issue, in this paper, we propose a self and parallax attention mechanism (SPAM) to aggregate the information from its own image and the counterpart stereo image simultaneously, thus reconstructing high-quality stereoscopic SR image pairs. Moreover, we design an efficient network architecture and effective loss functions to enforce stereo-consistency constraint. Finally, experimental results demonstrate the superiority of our method over state-of-the-art SR methods in terms of both quantitative metrics and qualitative visual quality while maintaining stereo-consistency between stereoscopic image pair.

Learning Parallax Attention for Stereo Image Super-Resolution

“…Wang et al [2], [3] proposed a parallax attention module to model stereo correspondence with a global receptive field along the epipolar line. Ying et al [4] proposed a stereo attention module and embedded it into pre-trained SISR networks for stereo image SR. Song et al [5] combined self-attention with parallax attention to propose a SPAMnet for stereo image SR. Furthermore, stereo consistency was addressed by SPAMnet [5] using disparity maps regressed from parallax attention maps.…”

“…Wang et al [2], [3] addressed the varying parallax issue by proposing a parallax attention module (PAM), and developed a PASSRnet for stereo image SR. Ying et al [4] addressed the information incorporation issue by equipping several stereo attention modules (SAMs) to the pretrained single image SR (SISR) networks. Song et al [5] addressed the occlusion issue by checking stereo consistency using disparity maps regressed by parallax attention maps. Although continuous improvements have been achieved, the inherent correlation within stereo image pairs are still under exploited, which hinders the performance improvement of stereo image SR methods.…”

Local-Global Fusion Network for Video Super-Resolution

“…Four deformable convolutions are utilized for frame alignment. The number of dilation convolutions in each DMDCU is [8,6,4,2] respectively. Both LFM and GFM have 10 internal blocks for feature fusion.…”

“…The first column represents the number of dilation convolutions in each DMDCU. For example, [8,6,4,2] indicates that the alignment module has four DCs and the number of dilation convolutions in each DMDCU is 8, 6, 4 and 2 respectively from shallow to deep DC. Also, the dilation rates range from 1 to 8, 1 to 6, 1 to 4 and 1 to 2 correspondingly.…”

“…0 means that in this DC we apply ordinary convolution rather than the DMDCU to predict the parameters. Through comparing [8,6,4,2] and [8,8,8,8] group, the usage of more dilation convolutions just promotes the performance slightly. The decreased strategy can avoid many redundant calculations so as to reduce the cost of space and time.…”