End-to-End View Synthesis for Light Field Imaging with Pseudo 4DCNN

Wang, Yunlong; Liu, Fei; Wang, Zhong Lin; Hou, Guangqi; Sun, Zhenan; Tan, Tieniu

doi:10.1007/978-3-030-01216-8_21

Cited by 115 publications

(108 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the above methods are applied on light field views, there also exist methods operating on epipolar plane images (EPI), in particular for angular interpolation or superresolution [4,5,6,11,16,19,20]. Focusing on learningbased solutions, Wu et al [19] model the light field reconstruction as a learning-based detail restoration in the EPIs.…”

Section: View Synthesis With Learned Epi Interpolationmentioning

confidence: 99%

Learning Fused Pixel and Feature-Based View Reconstructions for Light Fields

Shi

Jiang

Guillemot

2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

In this paper, we present a learning-based framework for light field view synthesis from a subset of input views. Building upon a lightweight optical flow estimation network to obtain depth maps, our method employs two reconstruction modules in pixel and feature domains respectively. For the pixel-wise reconstruction, occlusions are explicitly handled by a disparity-dependent interpolation filter, whereas inpainting on disoccluded areas is learned by convolutional layers. Due to disparity inconsistencies, the pixel-based reconstruction may lead to blurriness in highly textured areas as well as on object contours. On the contrary, the featurebased reconstruction well performs on high frequencies, making the reconstruction in the two domains complementary. End-to-end learning is finally performed including a fusion module merging pixel and feature-based reconstructions. Experimental results show that our method achieves state-of-the-art performance on both synthetic and realworld datasets, moreover, it is even able to extend light fields' baseline by extrapolating high quality views without additional training.

show abstract

Section: View Synthesis With Learned Epi Interpolationmentioning

confidence: 99%

Learning Fused Pixel and Feature-Based View Reconstructions for Light Fields

Shi

Jiang

Guillemot

2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…The method employs a disparity estimation network, a warping algorithm, and a color prediction network to synthesize a single view in the LF image. In [22], an end-to-end deep-learning-based view synthesis method is proposed based on a system of 2D convolutions applied to stacked epipolar plane images and of 3D convolutions for detail-restoration. In [23], the authors propose a lossy compression scheme based on depth image-based view synthesis technique, where four reference views are compressed by HEVC and used to reconstruct a cropped version of the LF image.…”

Section: I S T a T E -O F -T H E -A R Tmentioning

confidence: 99%

Deep-learning-based macro-pixel synthesis and lossless coding of light field images

Schiopu

Munteanu

2019

SIP

View full text Add to dashboard Cite

Deep-learning-based macro-pixel synthesis and lossless coding of light field imagesionut schiopu and adrian munteanu This paper proposes a novel approach for lossless coding of light field (LF) images based on a macro-pixel (MP) synthesis technique which synthesizes the entire LF image in one step. The reference views used in the synthesis process are selected based on four different view configurations and define the reference LF image. This image is stored as an array of reference MPs which collect one pixel from each reference view, being losslessly encoded as a base layer. A first contribution focuses on a novel network design for view synthesis which synthesizes the entire LF image as an array of synthesized MPs. A second contribution proposes a network model for coding which computes the MP prediction used for lossless encoding of the remaining views as an enhancement layer. Synthesis results show an average distortion of 29.82 dB based on four reference views and up to 36.19 dB based on 25 reference views. Compression results show an average improvement of 29.9 over the traditional lossless image codecs and 9.1 over the state-of-the-art.

show abstract

“…Others have deployed full 4D light fields [18,31], albeit at the cost of complex hardware setups and increased computational cost. Recently, deep learning techniques have been applied in similar settings to fill holes and eliminate artifacts caused by the sampling gap, dis-occlusions, and inaccurate 3D reconstructions [14,19,61,55,49,13,37]. While improving results over traditional methods, such approaches rely on multi-view input and are hence limited to the same setting.…”

Section: Related Workmentioning

confidence: 99%

Monocular Neural Image Based Rendering With Continuous View Control

Song

Chen

Hilliges

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Interactive novel view synthesis: Given a single source view our approach can generate a continuous sequence of geometrically accurate novel views under fine-grained control. Top: Given a single street-view like input, a user may specify a continuous camera trajectory and our system generates the corresponding views in real-time. Bottom: An unseen hires internet image is used to synthesize novel views, while the camera is controlled interactively. Please refer to our project homepage † .

show abstract

End-to-End View Synthesis for Light Field Imaging with Pseudo 4DCNN

Cited by 115 publications

References 23 publications

Learning Fused Pixel and Feature-Based View Reconstructions for Light Fields

Learning Fused Pixel and Feature-Based View Reconstructions for Light Fields

Deep-learning-based macro-pixel synthesis and lossless coding of light field images

Monocular Neural Image Based Rendering With Continuous View Control

Contact Info

Product

Resources

About