Learning Non-Local Spatial-Angular Correlation for Light Field Image Super-Resolution

Abstract

Exploiting spatial-angular correlation is crucial to light field (LF) image super-resolution (SR), but is highly challenging due to its non-local property caused by the disparities among LF images. Although many deep neural networks (DNNs) have been developed for LF image SR and achieved continuously improved performance, existing methods cannot well leverage the long-range spatialangular correlation and thus suffer a significant performance drop when handling scenes with large disparity variations. In this paper, we propose a simple yet effective method to learn the non-local spatial-angular correlation for LF image SR. In our method, we adopt the epipolar plane image (EPI) representation to project the 4D spatialangular correlation onto multiple 2D EPI planes, and then develop a Transformer network with repetitive self-attention operations to learn the spatial-angular correlation by modeling the dependencies between each pair of EPI pixels. Our method can fully incorporate the information from all angular views while achieving a global receptive field along the epipolar line. We conduct extensive experiments with insightful visualizations to validate the effectiveness of our method. Comparative results on five public datasets show that our method not only achieves state-of-the-art SR performance, but also performs robust to disparity variations. Code is publicly available at https://github.com/ ZhengyuLiang24/EPIT.

NTIRE 2022 Challenge on Stereo Image Super-Resolution: Methods and Results

“…It can be observed from Table 1 that all these methods surpass the state-of-the-art method DistgSSR [2], and 9 of them surpass the recent top-performing method EPIT [88]. Note that, the winner solution proposed by the OpenMeow achieves around 1 dB improvement in PSNR over Dist-gSSR [2] on both test and validation sets, which significantly push the state-of-the-art of LF image SR to a new height.…”

“…Transformers are used as the basic architecture in 6 solutions, while other models are purely based on CNNs. The idea of LF disentangling [2] was adopted in most solutions, and the recently developed method EPIT [88] was used as the backbone by the OpenMeow team (winner) and the BNU-AI-TRY team. Subspace division.…”

“…Wang et al [87] proposed a Multi-granularity Aggregation Transformer (MAT) for LF image SR, in which the LF feature representation was learned via three designed granularity aggregation units. More recently, Liang et al [88] investigated the non-local spatial-angular correlations in LF image SR, and developed a Transformer-based network called EPIT to achieve state-of-the-art SR performance. The proposed EPIT achieves a global receptive field along the epipolar line, and is robust to disparity variations.…”

“…This method is mainly inspired by the recent EPIT method [88], and aims to improve the capability of the spatial-angular correlation modeling. Specifically, this team…”

“…Training Set. This challenge follows the existing LF image SR works [2,71,74,84,85,88], and uses the EPFL [35], HCInew [36], HCIold [37], INRIA [38] and STFgantry [39] datasets for training. All the 144 LFs in the training set have an angular resolution of 9 × 9.…”