Stereo Analysis by Hybrid Recursive Matching for Real-Time Immersive Video Conferencing

Abstract

Real-time stereo analysis is an important research area in computer vision. In this context, we propose a stereo algorithm for an immersive video-conferencing system by which conferees at different geographical places can meet under similar conditions as in the real world. For this purpose, virtual views of the remote conferees are generated and adapted to the current viewpoint of the local participant. Dense vector fields of high accuracy are required in order to guarantee an adequate quality of the virtual views. Due to the usage of a wide baseline system with strongly convergent camera configurations, the dynamic disparity range is about 150 pixels. Considering computational costs, a full search or even a local search restricted to a small window of a few pixels, as it is implemented in many real-time algorithms, is not suitable for our application because processing on full-resolution video according to CCIR 601 TV standard with 25 frames per second is addressed-the most desirable as a pure software solution running on available processors without any support from dedicated hardware. Therefore, we propose in this paper a new fast algorithm for stereo analysis, which circumvents the window search by using a hybrid recursive matching strategy based on the effective selection of a small number of candidates. However, stereo analysis requires more than a straightforward application of stereo matching. The crucial problem is to produce accurate stereo correspondences in all parts of the image. Especially, errors in occluded regions and homogenous or less structured regions lead to disturbing artifacts in the synthesized virtual views. To cope with this problem, mismatches have to be detected and substituted by a sophisticated interpolation and extrapolation scheme.

Overview of Photogrammetric Measurement Techniques in Minimally Invasive Surgery Using Endoscopes

“…The interior and relative orientation was determined by a common computer vision approach using a checkerboard pattern. For dense image matching a modified census similarity measure and a hybrid recursive matching (HRM) algorithm (Atzpadin et al, 2004) was applied. The HRM requires the current stereo image pair and the previous disparity map to generate robust 3D point clouds in real-time (Figure 6 top).…”

An Uncalibrated View-Synthesis Pipeline

“…Once the images are rectified, dense matches can be obtained using any stereo matching algorithm, for example [9] or [4]. The correspondences are transferred back to the original reference images by applying the inverse of the rectifying transformation (de-rectification).…”

“…We are grateful to O. Schreer who ran his stereo matching code [4] on our images and provided us with the disparity. The use of the Matlab SIFT code by A. Vedaldi (www.cs.ucla.edu/vedaldi/sift/sift.htm) and the Matlab RANSAC code by P. Kovesi (www.csse.uwa.edu.au/˜pk/research/matlabfns/) is here acknowledged.…”

A Flexible 3D TV System for Different Multi-Baseline Geometries

“…For disparity estimation, we have chosen a fast HRM (Hybrid Recursive Matching) algorithm [6]. Due to its recursive structure, the HRM algorithm produces extremely smooth and temporally consistent "per-pixel" disparity maps.…”