Automatic Real-Time Video Matting Using Time-of-Flight Camera and Multichannel Poisson Equations

Abstract

This paper presents an automatic real-time video matting system. The proposed system consists of two novel components. In order to automatically generate trimaps for live videos, we advocate a Time-of-Flight (TOF) camerabased approach to video bilayer segmentation. Our algorithm combines color and depth cues in a probabilistic fusion framework. The scene depth information returned by the TOF camera is less sensitive to environment changes, which makes our method robust to illumination variation, dynamic background and camera motion. For the second step, we perform alpha matting based on the segmentation Electronic supplementary material The online version of this article (doi:10.1007/s11263-011-0471-x) contains supplementary material, which is available to authorized users. result. Our matting algorithm uses a set of novel Poisson equations that are derived for handling multichannel color vectors, as well as the depth information captured. Realtime processing speed is achieved through optimizing the algorithm for parallel processing on graphics hardware. We demonstrate the effectiveness of our matting system on an extensive set of experimental results.

Overview of efficient high-quality state-of-the-art depth enhancement methods by thorough design space exploration

“…Figure 6 qualitatively compares JP, JPMC, 3DJBU and JPMC on real depth images acquired by a time-of-flight (ToF) depth camera. For this experiment, we use the depth sequence ms provided by Wang et al [46] which is available for download from [40]. This sequence consist of synchronized 320 by 240 RGB color and depth images with the same resolution that have been recorded by ZCam from 3DV Systems.…”

High-Accuracy and Quick Matting Based on Sample-Pair Refinement and Local Optimization

“…Recent methods [21]- [23] utilized the depth image to bi-layer segment the image, followed by morphology operations on boundaries to generate the trimap automatically. In our experiments we employ the Kinect depth camera [24] to capture images.…”

Fusion of TLS and RGB point clouds with TIR images for indoor mobile mapping

“…Coregistration for TOF cameras and RGB images is done calculating the relative orientation in a bundle adjustment with homologous points [11] due to the fact, that the radiometric behaviour in near infrared and visible light is almost the same. Wang et al [12] investigate foreground background separation from combining TOF depth values and RGB values, both recorded by one camera system. Due to the RGB camera, they learn a likelihood classification for foreground and background colors.…”

“…In case of a thermal camera, the temperature of a person is known and so a fixed threshold can be used instead. In contrast to [12] a more complex geometric calibration has to be done for TOF and TIR cameras as to different optics are used an so a relative orientation has to be calculated [13].…”