Spatio-Temporally Consistent View Synthesis From Video-Plus-Depth Data With Global Optimization

Hsu, Hsiao-An; Chiang, Chen-Kuo; Lai, Shang-Hong

doi:10.1109/tcsvt.2013.2276699

Cited by 17 publications

(16 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yet, these methods are prone to exhibit spatio-temporal inconsistencies for moving foreground objects. Hsu et al proposed a method to construct the disocclusions using global optimization [36]. However, the results show spatial inconsistencies at the structured background due to the inpainting method used.…”

Section: Related Workmentioning

confidence: 99%

Spatio-temporal consistent depth-image-based rendering using layered depth image and inpainting

Muddala

Olsson

Sjöström

2016

J Image Video Proc.

View full text Add to dashboard Cite

Depth-image-based rendering (DIBR) is a commonly used method for synthesizing additional views using video-plus-depth (V+D) format. A critical issue with DIBR-based view synthesis is the lack of information behind foreground objects. This lack is manifested as disocclusions, holes, next to the foreground objects in rendered virtual views as a consequence of the virtual camera "seeing" behind the foreground object. The disocclusions are larger in the extrapolation case, i.e. the single camera case. Texture synthesis methods (inpainting methods) aim to fill these disocclusions by producing plausible texture content. However, virtual views inevitably exhibit both spatial and temporal inconsistencies at the filled disocclusion areas, depending on the scene content. In this paper, we propose a layered depth image (LDI) approach that improves the spatio-temporal consistency. In the process of LDI generation, depth information is used to classify the foreground and background in order to form a static scene sprite from a set of neighboring frames. Occlusions in the LDI are then identified and filled using inpainting, such that no disocclusions appear when the LDI data is rendered to a virtual view. In addition to the depth information, optical flow is computed to extract the stationary parts of the scene and to classify the occlusions in the inpainting process. Experimental results demonstrate that spatio-temporal inconsistencies are significantly reduced using the proposed method. Furthermore, subjective and objective qualities are improved compared to state-of-the-art reference methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Spatio-temporal consistent depth-image-based rendering using layered depth image and inpainting

Muddala

Olsson

Sjöström

2016

J Image Video Proc.

View full text Add to dashboard Cite

show abstract

“…Both the color image and the depth map are warped using the method from [18]. The warped images contain nondefined (missing) regions, which can be classified into two types: cracks, caused by noise or digital numerical precision, and disocclusions, caused by sharp depth discontinuities [16] (see the black areas in Figs. 2(c) and (d)).…”

Section: A Image Warpingmentioning

confidence: 99%

“…In literature, several depth map filling approaches were proposed [11], [16], [19]. We have tested some of these methods and concluded that the histogram-based method from [19] performs the best, thus we adopted it in our approach.…”

Section: Depth Map Fillingmentioning

confidence: 99%

“…the background constraint should be enforced. For that reason, several disocclusion filling methods have been proposed [3], [10]- [16], which adapt the general inpainting strategy to this specific problem by incorporating depth information.…”

Section: Introductionmentioning

confidence: 99%

“…Additionally, a depth map filling method based on K-means clustering of depth values is proposed. A very recent method from [16] uses a similar view synthesis framework as [3], but introduces a global MRF-based method for spatio-temporally consistent disocclusion filling. Labels are selected based on the combination of depth, structure and texture information and the resulting optimization problem is solved with a coarseto-fine belief propagation strategy to efficiently deal with the huge number of labels, similarly to [17].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Depth-guided patch-based disocclusion filling for view synthesis via Markov random field modelling

Ružić

Jovanov

Luong

et al. 2014

2014 8th International Conference on Signal Processing and Communication Systems (ICSPCS)

View full text Add to dashboard Cite

In this paper, we propose a novel patch-based disocclusion filling method for view synthesis from video-plusdepth data. The proposed method treats disocclusion filling as a global optimization problem, where global (spatial) consistency among the patches is enforced via a Markov random field (MRF) model. The main idea of our method is to exploit disocclusion properties to limit and guide the search for candidate patches (labels) and to minimize efficiently the resulting MRF energy. In particular, we propose to constrain the label selection to local background regions in order to ensure that the disocclusions are filled with background information. Background is determined based on the locally estimated hard threshold on the depth values. The efficient minimization approach represents an extension of our previous method for general inpainting, where we propose to visit the MRF nodes from the background to the foreground disocclusion border and discard unnecessary labels. In this way, the number of labels is further reduced and the propagation of background information is additionally enforced. Finally, efficient inference is performed to obtain the final inpainting result. The proposed disocclusion filling method represents one step of the complete view synthesis framework that we also introduce in this paper. Experimental results show improvement of the proposed approach over related state-of-the-art methods both for small and big disocclusions.

show abstract

Video Stabilization via Prediction with Time‐Series Network and Image Inpainting with Pyramid Fusion

Cheng

Lan

et al. 2021

Chinese J of Electronics

View full text Add to dashboard Cite

Due to the poor filling effect of the video image defect commonly used in the video stabilization field, the video is seemed still unstable after the image stabilization process, which seriously affects the visual effect. To solve this problem, we improve a video stabilization method based on time‐series network prediction and pyramid fusion restoration is proposed to optimize the visual effect after image stabilization. The flow of the proposed method is as follows: First, it is adaptive to determine whether the defect of the corresponding frame at the current time needs padding inpainting. Then, for the frame that needs to be inpainting, the frames generated before the current moment are sent to the model combining the convolutional neural networks and the gate recurrent unit to predict the part to be filled. Next the current defect image and the complete image to be filled are brought into the Laplacian pyramid reconstruction, and the improved weighted optimal suture is introduced for splicing during the fusion. Finally, the video frame is cut after reconstruction. The method is tested on a data set composed of videos commonly used in the field of video stabilization. The experimental results show that the average peak signal to noise ratio of the method is 2 to 5dB higher than that of the comparison algorithm, and the average structural similarity index is improved by about 2% to 7% compared with the contrast algorithm.

show abstract

Spatio-Temporally Consistent View Synthesis From Video-Plus-Depth Data With Global Optimization

Cited by 17 publications

References 19 publications

Spatio-temporal consistent depth-image-based rendering using layered depth image and inpainting

Spatio-temporal consistent depth-image-based rendering using layered depth image and inpainting

Depth-guided patch-based disocclusion filling for view synthesis via Markov random field modelling

Video Stabilization via Prediction with Time‐Series Network and Image Inpainting with Pyramid Fusion

Contact Info

Product

Resources

About