Layered approach for improving the quality of free-viewpoint depth-image-based rendering images

Smirnov, Sergey; Battisti, Federica; Gotchev, Atanas

doi:10.1117/1.jei.28.1.013049

Cited by 5 publications

(3 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Han et al [22] used the multi-threshold Otsu method to segment the depth image into multiple layers and performed layered 3D warping. In [23], threshold segmentation is used to extract foreground object and the background layer is compensated. In [24], disocclusion edge pixels are divided into foreground and background based on the depth value.…”

Section: Related Workmentioning

confidence: 99%

Local Foreground Removal Disocclusion Filling Method for View Synthesis

Liang

Chen

et al. 2020

IEEE Access

View full text Add to dashboard Cite

View synthesis is an effective method to generate the contents of multiple views based on a limited number of reference views, which can be used in 2D to 3D conversion, free viewpoint video and multiview video rendering. Depth-image-based rendering (DIBR) is a practical technique to generate virtual view by using a 2D reference view and its depth image. However, a critical problem in DIBR process is that disocclusions might be produced in the synthesized image because the background occluded by the foreground objects in the reference view may be exposed in the virtual view. In this paper, a local foreground removal method is proposed for disocclusion filling. Morphology-based depth image preprocessing is performed before DIBR, aiming to correct the depth value of the ghosts and remove ghost artifacts. In the synthesized virtual image, pixels on the disocclusion edge are identified and classified. Then they are positioned in the reference image by inverse 3D warping. Local foreground regions that occlude the corresponding background are removed from both the reference image and its depth image based on the disocclusion edge pixels. Removed region is filled with surrounding background contents, and depth information is used in this process to prevent foreground penetration. The predicted background contents are warped to the disocclusion region, thereby achieving the hole filling. Experimental results show that the proposed method performs better than the other methods in disocclusion filling, and improves the subjective and objective quality of the synthesized view. In the evaluation results of PSNR, SSIM, FSIMc and VSI, our method improves by 0.

show abstract

Section: Related Workmentioning

confidence: 99%

Local Foreground Removal Disocclusion Filling Method for View Synthesis

Liang

Chen

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…View synthesis aims to create novel views of an object or a scene from a perspective of a virtual camera based on a set of reference images. It has been an active field of research already for several decades in computer vision and computer graphics due to its various application areas including free-viewpoint television, virtual and augmented reality, and telepresence [1,2,3,4,5].…”

Section: Introductionmentioning

confidence: 99%

“…This allows for larger baselines between the views, but also sets high requirements to the quality of the 3D model. Between these two extremes, free viewpoint depth-image-based rendering (DIBR) uses depth maps associated to the reference views enabling 3D image warping to synthesize the novel view [3]. In practice, all these approaches tend to produce notable artifacts due to missing or inaccurate data, which reduced the quality of the rendered image.…”

Section: Introductionmentioning

confidence: 99%

Sequential View Synthesis with Transformer

Nguyen-Ha

Huynh

Rahtu

et al. 2021

Computer Vision – ACCV 2020

View full text Add to dashboard Cite

This paper addresses the problem of novel view synthesis by means of neural rendering, where we are interested in predicting the novel view at an arbitrary camera pose based on a given set of input images from other viewpoints. Using the known query pose and input poses, we create an ordered set of observations that leads to the target view. Thus, the problem of single novel view synthesis is reformulated as a sequential view prediction task. In this paper, the proposed Transformer-based Generative Query Network (T-GQN) extends the neural-rendering methods by adding two new concepts. First, we use multi-view attention learning between context images to obtain multiple implicit scene representations. Second, we introduce a sequential rendering decoder to predict an image sequence, including the target view, based on the learned representations. Finally, we evaluate our model on various challenging datasets and demonstrate that our model not only gives consistent predictions but also doesn't require any retraining for finetuning.

show abstract