42.1: A Real‐time 3D Multi‐view Rendering From a Real‐time 3D Capture

Abstract

As often declared by customers, wearing glasses is a clear limiting factor for 3D adoption in the home. Auto-stereoscopic systems bring an interesting answer to this issue. These systems are evolving very fast providing improved picture quality. Nevertheless, they require adapted content which are today highly demanding in term of computation power. The system we describe here is able to generate in real-time adapted content to these displays: from a real-time stereo capture up to a real-time multi-view rendering. A GPU-based solution is proposed that ensures a real-time processing of both disparity estimation and multi-view rendering on the same platform. IntroductionAs often declared by customers, wearing glasses is a clear limiting factor for 3D adoption in the home. Such glasses can be heavy, they can easily been broken, their battery needs to be recharged. These are common remarks made on active glasses. Although passive glasses relax a bit these constraints, there are still reluctances to wear specific 3D glasses at home. Multi-view displays will take rapidly benefit of it.As it is often the case with new display technologies, the question of content is key. Multi-view displays require in theory a multiview acquisition with a large number of cameras. As it is technically and economically not feasible, Depth Image Based Rendering techniques are used to generate the required number of views [1]. For digital signage application, without the real-time constraint, the generation of such interpolated views is not a critical point. As soon as real-time is a constraint, then achieving a correct DIBR is a challenge. The generation of good disparity map associated with an efficient multi-view rendering is a critical task. This is the purpose of this paper to describe the system we put in place to realize a convincing real-time multi-view rendering from a real-time 3D capture.In the section 2, we describe the complex algorithm behind the technology put in place. The section 3 is dedicated to the real-time implementation, in particular the use of a GP-GPU processor. In section 4 we describe the results we got and we finally conclude the paper with some perspectives. AlgorithmAs described in section 1, the system presented here can be split into several blocks. For some of them a dedicated algorithm study has been performed to reach the expected level of quality. The main blocks described here are the disparity estimation, the view interpolation and the specific display sub-pixel shuffling. Disparity estimationClassical dense binocular stereo vision computes a disparity map between two rectified images of the same scene supposed to be Lambertian where the value at each pixel in the disparity map is the estimated shift between corresponding points. Matching corresponding points in a pair of stereo images is an ill-posed problem. Indeed, the matching process can be ambiguous due to noise in the images, poorly textured regions, repeated structures (Strob effect) and occlusions (points appearing in only one of the i...