Adaptive Image Warping for Hole Prevention in 3D View Synthesis

Abstract

Increasing popularity of 3D videos calls for new methods to ease the conversion process of existing monocular video to stereoscopic or multi-view video. A popular way to convert video is given by depth image-based rendering methods, in which a depth map that is associated with an image frame is used to generate a virtual view. Because of the lack of knowledge about the 3D structure of a scene and its corresponding texture, the conversion of 2D video, inevitably, however, leads to holes in the resulting 3D image as a result of newly-exposed areas. The conversion process can be altered such that no holes become visible in the resulting 3D view by superimposing a regular grid over the depth map and deforming it. In this paper, an adaptive image warping approach as an improvement to the regular approach is proposed. The new algorithm exploits the smoothness of a typical depth map to reduce the complexity of the underlying optimization problem that is necessary to find the deformation, which is required to prevent holes. This is achieved by splitting a depth map into blocks of homogeneous depth using quadtrees and running the optimization on the resulting adaptive grid. The results show that this approach leads to a considerable reduction of the computational complexity while maintaining the visual quality of the synthesized views.

Efficient Hybrid Image Warping for High Frame-Rate Stereoscopic Rendering

“…While this popular method is cost-efficient, it introduces noticeable artifacts especially in the presence of high frequencies in the depth buffer. In order to reduce these artifacts, several works perform a low-pass filtering on the depth-buffer [10,19,20,32].…”

Spatio-temporal consistent depth-image-based rendering using layered depth image and inpainting

“…However, post-processing techniques add complicated steps after warping in the warped image [6][7][8][9]. Image-domain warping methods are another way of reconstructing the virtual views using sparse disparity information [10,11]. A number of methods combine both pre-processing and post-processing techniques to address rendering problems [8].…”

“…(10). Next, the moving object pixels and foreground pixels in the sprite are replaced by information in the new frame:…”

Panorama View With Spatiotemporal Occlusion Compensation for 3D Video Coding

“…A patch around the hole is selected and searched for in the rest of the image; then the best match is used to recover the occlusion [32]- [34]. In depth based inpainting either the depth map is inpainted first and then used to find (by 3D warping) the pixels that can cover the holes or the recovered depth map is used in conjunction with texture-based or patch-based inpainting [35]- [38].…”