3D modeling and motion parallax for improved videoconferencing

Abstract

We consider a face-to-face videoconferencing system that uses a Kinect camera at each end of the link for 3D modeling and an ordinary 2D display for output. The Kinect camera allows a 3D model of each participant to be transmitted; the (assumed static) background is sent separately. Furthermore, the Kinect tracks the receiver's head, allowing our system to render a view of the sender depending on the receiver's viewpoint. The resulting motion parallax gives the receivers a strong impression of 3D viewing as they move, yet the system only needs an ordinary 2D display. This is cheaper than a full 3D system, and avoids disadvantages such as the need to wear shutter glasses, VR headsets, or to sit in a particular position required by an autostereo display. Perceptual studies show that users experience a greater sensation of depth with our system compared to a typical 2D videoconferencing system.

Inpainting of Wide-Baseline Multiple Viewpoint Video

“…Multiple viewpoint video (MVV) inpainting has received little attention. Literature addresses only the narrow baseline case, typically that of stereo pairs [16,25,32,34,35,40]. Given the small extent of the area to be filled, approaches approximate missing content by extrapolating adjacent colourdepth information and performing patch matching over the extrapolated pixels.…”

Static Scene Illumination Estimation from Videos with Applications

“…The geometry of a scene in the image can be approximated by user interaction [2,[21][22][23][24] , inferred by scene understanding algorithms [25][26] or using depth sensors [27] . Many algorithms can directly estimate the geometry of the scene from a single image using very simple structures [28][29][30] .…”

Practical automatic background substitution for live video

“…It has extensive applications in video composition [1,2], video conferencing [3,4], and augmented reality [5]. The process of background substitution can be basically separated into two steps.…”