Pushing the Boundaries of View Extrapolation With Multiplane Images

Srinivasan, Pratul P.; Tucker, Richard; Barron, Jonathan T.; Ramamoorthi, Ravi; Ng, Ren; Snavely, Noah

doi:10.1109/cvpr.2019.00026

Cited by 271 publications

(113 citation statements)

References 43 publications

(43 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A great amount of effort has been dedicated to incorporate geometric information to the model. For example, [18,19,20,21,22,23] apply deep learning techniques to leverage geometry cues and learn to predict the novel view. The deep learning based-light field camera view interpolation [20,24] use a deep network to predict depth separately for every novel view.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Sequential View Synthesis with Transformer

Nguyen-Ha

Huynh

Rahtu

et al. 2021

Computer Vision – ACCV 2020

View full text Add to dashboard Cite

This paper addresses the problem of novel view synthesis by means of neural rendering, where we are interested in predicting the novel view at an arbitrary camera pose based on a given set of input images from other viewpoints. Using the known query pose and input poses, we create an ordered set of observations that leads to the target view. Thus, the problem of single novel view synthesis is reformulated as a sequential view prediction task. In this paper, the proposed Transformer-based Generative Query Network (T-GQN) extends the neural-rendering methods by adding two new concepts. First, we use multi-view attention learning between context images to obtain multiple implicit scene representations. Second, we introduce a sequential rendering decoder to predict an image sequence, including the target view, based on the learned representations. Finally, we evaluate our model on various challenging datasets and demonstrate that our model not only gives consistent predictions but also doesn't require any retraining for finetuning.

show abstract

Section: Related Workmentioning

confidence: 99%

“…The deep learning based-light field camera view interpolation [20,24] use a deep network to predict depth separately for every novel view. Another line of work [18,19,21] cleverly extract a Multiplane image representation of the scene. This representation offers regularization that allows for an impressive stereo baseline extrapolation.…”

Section: Related Workmentioning

confidence: 99%

Sequential View Synthesis with Transformer

Nguyen-Ha

Huynh

Rahtu

et al. 2021

Computer Vision – ACCV 2020

View full text Add to dashboard Cite

show abstract

“…Soft3D introduced a view synthesis pipeline to synthesize accurate images, even in the occlusion areas, by a visibility refinement process. The original work was cited in many follow-on view synthesis investigations [ 27 , 28 , 29 ]. However, few investigations were introduced to iteratively update the PSS matching costs for the purpose of obtaining accurate 3D depth reconstruction.…”

Section: Previous Workmentioning

confidence: 99%

“…The iterative enhancement scheme of the network is similar to the refinement of the soft visibility of Soft3D. Srinivasan et al [ 29 ] introduce a view extrapolation method using the MPI training network. This network employs a 3D convolutional network to generate and train occlusion-free MPI from PSS volumes as network inputs.…”

Section: Previous Workmentioning

confidence: 99%

Enhanced Soft 3D Reconstruction Method with an Iterative Matching Cost Update Using Object Surface Consensus

Lee

Yun

et al. 2021

Sensors

View full text Add to dashboard Cite

In this paper, we propose a multi-view stereo matching method, EnSoft3D (Enhanced Soft 3D Reconstruction) to obtain dense and high-quality depth images. Multi-view stereo is one of the high-interest research areas and has wide applications. Motivated by the Soft3D reconstruction method, we introduce a new multi-view stereo matching scheme. The original Soft3D method is introduced for novel view synthesis, while occlusion-aware depth is also reconstructed by integrating the matching costs of the Plane Sweep Stereo (PSS) and soft visibility volumes. However, the Soft3D method has an inherent limitation because the erroneous PSS matching costs are not updated. To overcome this limitation, the proposed scheme introduces an update process of the PSS matching costs. From the object surface consensus volume, an inverse consensus kernel is derived, and the PSS matching costs are iteratively updated using the kernel. The proposed EnSoft3D method reconstructs a highly accurate 3D depth image because both the multi-view matching cost and soft visibility are updated simultaneously. The performance of the proposed method is evaluated by using structured and unstructured benchmark datasets. Disparity error is measured to verify 3D reconstruction accuracy, and both PSNR and SSIM are measured to verify the simultaneous enhancement of view synthesis.

show abstract

“…The multi-plane image (MPI) representation is another way to synthesize a 3D image that enables to render each pixel to get scene independent new views with consistent non-occlusion when multiple objects are involved in the scene. Each image plane is considered as a RGBA image belongs to the part of frustum with apex at lens and positioned at fixed equally spaced depth obtained as inverse of disparity [22], [23], [24], [25], [26], [27]. Recently, we have utilized MPIs as L number of fronto-parallel planes for synthesizing 3D perception image corresponding to L number of MRIs, and each MPIs are positioned in the DoFo zone belonging to image space with non-uniform interval using respective inter-depths [14].…”

Section: B Review On Layered Representation For 3d Perception Imagementioning

confidence: 99%

Accuracy in Depth Recovery and 3D Image Synthesis From Single Image Using Multi-Color Filter Aperture and Shallow Depth of Field

Deshpande

Bhatt²,

Madhavi³

2021

IEEE Access

View full text Add to dashboard Cite

A computational 3D image generation using a single view with multi-color filter aperture (MCA) and multi-plane representation is a cost-effective approach and most useful when there is no option to acquire either stereo or multi-views with orientation at all. Although this approach generates 3D perception image that includes multiple objects with both similar and dissimilar colors having occluded by each other, it may be insufficient for virtual/augmented reality applications due to inaccurate depth. In this article, we obtain a more accurate geometric depth estimation by formulating a suitable relationship between inter-objects depth of the 3D scene in the depth-of-field (DoF) zone and its corresponding inter-image plane depths of a 3D perception image in depth-of-focus (DoFo) zone of a given camera under shallow DoF zone constraint. But, this shallow depth zone is configured to be dependent only on the focal distance between the lens and object while the remaining parameters such as aperture diameter, focal length, and sensor sensitivity are held at constant values. All-in-focus 3D perception image is synthesized from multi-plane images (MPIs) by utilizing the inter-image plane depths computed from the disparities caused across the boundaries and its smooth surface from image textures inside the respective boundaries of the 2D MCA image. The 2.1D sketch is used as a semantic segmentation technique to determine the number of objects in the 3D scene as one in-focus region and the rest as out-of-focus regions due to the circle of confusions (CoCs) on the fixed image sensor plane. The same enables both ordering of the image regions and identifying occlusion wherever applicable. An accurate depth 3D image is synthesized, replacing accurate inter-depths in place of inter-depth between MPIs used for 3D perception image. In the end, the paper summarizes few experimental validations for the proposed approach with some salient examples having depth gaps between 0.5cm to 10.5cm. INDEX TERMS2D Object plane, 2D Image sensor plane, 2.1D Sketch, 3D Perception image, 3D Virtual object, Accurate depth, Computational 3D imaging, Depth of field, Depth of focus, Disparity, Multi-color filter aperture(M CA), Multi-region imaage (M RI) , Multi-plane image (M P I).

show abstract

Pushing the Boundaries of View Extrapolation With Multiplane Images

Cited by 271 publications

References 43 publications

Sequential View Synthesis with Transformer

Sequential View Synthesis with Transformer

Enhanced Soft 3D Reconstruction Method with an Iterative Matching Cost Update Using Object Surface Consensus

Accuracy in Depth Recovery and 3D Image Synthesis From Single Image Using Multi-Color Filter Aperture and Shallow Depth of Field

Contact Info

Product

Resources

About