Fast: Flow-Assisted Shearlet Transform for Densely-Sampled Light Field Reconstruction

Abstract

The Image-Based Rendering (IBR) approach using Shearlet Transform (ST) is one of the most effective methods for Densely-Sampled Light Field (DSLF) reconstruction. The ST-based DSLF reconstruction typically relies on an iterative thresholding algorithm for Epipolar-Plane Image (EPI) sparse regularization in shearlet domain, involving dozens of transformations between image domain and shearlet domain, which are in general time-consuming. To overcome this limitation, a novel learning-based ST approach, referred to as Deep Residual Shearlet Transform (DRST), is proposed in this paper. Specifically, for an input sparsely-sampled EPI, DRST employs a deep fully Convolutional Neural Network (CNN) to predict the residuals of the shearlet coefficients in shearlet domain in order to reconstruct a densely-sampled EPI in image domain. The DRST network is trained on synthetic Sparsely-Sampled Light Field (SSLF) data only by leveraging elaborately-designed masks. Experimental results on three challenging real-world light field evaluation datasets with varying moderate disparity ranges (8 -16 pixels) demonstrate the superiority of the proposed learning-based DRST approach over the non-learning-based ST method for DSLF reconstruction. Moreover, DRST provides a 2.4x speedup over ST, at least.Index Terms-Densely-sampled light field reconstruction, novel view synthesis, epipolar-plane image, Shearlet Transform (ST), Deep Residual Shearlet Transform (DRST).• We propose a learning-based DRST method that achieves better DSLF reconstruction performance than the nonlearning-based ST algorithm on three evaluation datasets arXiv:2003.08865v1 [cs.MM]

Light Field Reconstruction With a Hybrid Sparse Regularization-Pseudo 4DCNN Framework

“…Vagharshakyan et al [14], [15] proposed an approach using the sparse representation of EPIs in the shearlet transform domain. Gao et al further introduced optic flow [36], [37] and video frame interpolation methods [38] to improve the reconstruction result of shearlets-based methods. However, shearlet transform requires a precise estimation of the disparity range of the undersampled LF in order to design a shearlet system with decent scales and to pre-shear the undersampled EPIs.…”

Large-Scale Light Field Capture and Reconstruction

“…To overcome this limitation, this subsection will introduce a universal solution to the 3D DSLF reconstruction problem using optical flow techniques. Specifically, inspired by the success of Super-SloMo [JSJ+18] in video frame interpolation, a novel learning-based method, referred to as FAST [GKB+19a], is proposed to reconstruct 3D DSLFs from 3D SSLFs. The FAST method adopts one of the state-of-the-art optical flow approaches, i. e., PWC-Net [SYL+18], to estimate the bidirectional optical flow between neighboring views in an input SSLF.…”