Coherent spatial and temporal occlusion generation

Abstract

A vastly growing number of productions from the entertainment industry are aiming at 3D movie theatres. These productions use a two-view format, primarily intended for eye-wear assisted viewing in a well defined environment. To get this 3D content into the home environment, where a large variety of 3D viewing conditions exists (e.g different display sizes, display types, viewing distance), we need a flexible 3D format that can adjust the depth effect. Such a format is the image plus depth format in which a video frame is enriched with depth information of all pixels in the video. This format can be extended with an additional layer for occluded video and associated depth, that contains what is behind objects in the video. To produce 3D content in this extended format, one has to deduce what is behind objects. There are various axes along which this occluded data can be obtained. This paper presents a method to automatically detect and fill the occluded areas exploiting the temporal axis. To get visually pleasing results, it is of utmost importance to make the inpainting globally consistent. To do so, we start by analyzing data along the temporal axis and compute a confidence for each pixel. Then pixels from the future and the past that are not visible in the current frame are weighted and accumulated based on computed confidences. These results are then fed to a generic multi-source framework that computes the occlusion layer based on the available confidences and occlusion data.

Stereo video inpainting

“…Some artifacts from the temporal output are then filtered out by averaging with the other sources, while some of the resolution obtained by the temporal technique is lost due to mixing with a poor spatial reconstruction. Consequently the generated occlusion data (using the same terminology as Gunnewiek et al 6 ) is similar to a low-pass version of the optimal reconstruction. Similar to this work we propose to include all consistency constraints in a single framework, but we achieve reconstruction by copying best candidates from selected examples instead of merging several sources.…”

“…[2][3][4][5][6] However there has been little consideration of the problem in stereo, and as the production of stereoscopic content is rapidly increasing, so is the need for new tools to manipulate that content. Apart from object removal, applications of stereo inpainting include view synthesis, image-based modeling, occlusion generation for stereoscopic displays, 6 replacement of saturated areas in one of the two views of a stereo video 7 (due to differences in camera set-up), etc.…”

Spatio-temporal consistent depth-image-based rendering using layered depth image and inpainting

“…However, since motion vectors are sparse and do not correspond to the real motion, quality gains are limited. In other attempts, background sprites are generated by using the temporal frame information, after which disocclusions are filled and updated using the true texture and finally, inpainting is applied to fill holes in virtual views [29][30][31][32]. The method provided by Schmeing et al requires a spatial hole filling for disocclusions caused by static foregrounds [30].…”

“…where τ t D is the inter-frame average absolute difference at holes in frame t, N f is the number of frames, t is the number of hole pixels in the frame t and the flicker measured between the frames is given as (29) where τ t Dg and τ t Dv represent the inter-frame average absolute difference at holes of the ground truth and the virtual view at frame t, respectively. A lower F-score indicates less flickering.…”

Stereo-video inpainting

“…[2][3][4][5][6] Even so, there has been little consideration of the problem in stereo, and as the production of stereoscopic content is rapidly increasing, so is the need for new tools to manipulate that content. Manually touching up stereo videos would require at least twice as much work, not to mention taking into account consistency across views.…”