High-Order Residual Network for Light Field Super-Resolution

Abstract

Plenoptic cameras usually sacrifice the spatial resolution of their SAIs to acquire geometry information from different viewpoints. Several methods have been proposed to mitigate such spatio-angular trade-off, but seldom make use of the structural properties of the light field (LF) data efficiently. In this paper, we propose a novel high-order residual network to learn the geometric features hierarchically from the LF for reconstruction. An important component in the proposed network is the high-order residual block (HRB), which learns the local geometric features by considering the information from all input views. After fully obtaining the local features learned from each HRB, our model extracts the representative geometric features for spatio-angular upsampling through the global residual learning. Additionally, a refinement network is followed to further enhance the spatial details by minimizing a perceptual loss. Compared with previous work, our model is tailored to the rich structure inherent in the LF, and therefore can reduce the artifacts near non-Lambertian and occlusion regions. Experimental results show that our approach enables high-quality reconstruction even in challenging regions and outperforms state-of-the-art single image or LF reconstruction methods with both quantitative measurements and visual evaluation.

Light Field Image Super-Resolution Using Deformable Convolution

“…LF images captured by different devices (especially camera arrays) usually have significantly different baseline lengths. It is therefore, necessary to know how existing LF algorithms work under baseline variations, including those developed for depth estimation [73]- [78], view synthesis [79]- [87], and image SR [56], [88]- [91]. However, all existing LF datasets [16], [69]- [72] only include images with fixed baselines.…”

LightGAN: A Deep Generative Model for Light Field Reconstruction

“…The second category makes the use of deep learning frameworks. With the development of CNNs, backed by having more and more light field datasets [31], [32], some learning-based methods have been proposed recently and show promising performance [33], [34]. Kalantari et al [35] introduce the CNNs to traditional pipeline and produce plausible images.…”

“…The result is that these approaches have trouble approximating the correct EPIs of occluded or non-Lambertian regions, which therefore causes ghosting or tearing artifacts near the object boundaries. By contrast, the HDC layer shows its potential in processing the light field image for multiple applications, such as material recognition [16], view synthesis [38], and super-resolution [34]. These achievements are owed greatly to its ability to extract spatial representations preserving the angular correlations.…”

Light field Rectification based on relative pose estimation

“…Unlike the classic pinhole camera, an LF camera can capture both spatial and angular information of rays in the environment by a single 2D snapshot, which is the reason why the LF camera has gained significant interest in computer vision. Typical applications of the LF camera include simultaneous localization and mapping (SLAM) [3,4], 3D reconstruction [5][6][7], digital refocusing [8], virtual viewpoint synthesis [9], super resolution [10,11], etc.…”