Live Tracking and Dense Reconstruction for Handheld Monocular Endoscopy

Abstract

Contemporary endoscopic Simultaneous Localization And Mapping (SLAM) methods accurately compute endoscope poses, however, they only provide a sparse 3D reconstruction that poorly describes the surgical scene. We propose a novel dense SLAM method whose qualities are: 1) Monocular, requiring only RGB images of a hand-held monocular endoscope. 2) Fast, providing endoscope positional tracking and 3D scene reconstruction, running in parallel threads. 3) Dense, yielding an accurate dense reconstruction. 4) Robust, to the severe illumination changes, poor texture and small deformations that are typical in endoscopy. 5) Self-contained, without needing any fiducials nor external tracking devices, therefore it can be smoothly integrated into the surgical workflow. It works as follows. Firstly, accurate cluster frame poses are estimated using the sparse SLAM feature matches. The system segments clusters of video frames according to a parallax criteria. Next, dense matches between cluster frames are computed in parallel by a variational approach that combines Zero Mean Normalized Cross Correlation (ZNCC) and a gradient Huber norm regularizer. This combination copes with challenging lighting and textures at an affordable time budget on a modern GPU. It can outperform pure stereo reconstructions because the frames cluster can provide larger parallax from the endoscope's motion. We provide an extensive experimental validation on real sequences of the porcine abdominal cavity, both in-vivo and exvivo. We also show a qualitative evaluation on human liver. Additionally, we show a comparison with other dense SLAM methods showing the performance gain in terms of accuracy, density and computation time.

Whole Stomach 3D Reconstruction and Frame Localization From Monocular Endoscope Video

“…In this paper, we have presented an SfM pipeline to reconstruct the whole shape of a stomach from a standard monocular endoscope video. For this work, we have decided to adopt SfM because it has numbers of advantages compared to other approaches such as SfS [11]- [13] and SLAM [14]- [17]. The SfS can recover the 3D structure from a single image.…”

“…In addition, we have found that red channel images under the chromo-endoscopy using the IC dye provides the most complete point cloud result. For comparison, we run the base version of SLAM [39] applied in [17] on Subject B. Even on the red channel with the IC dye, the SLAM cannot obtain enough feature matches to maintain the feature tracking, resulting in the incomplete 3D model far from the whole stomach with very few reconstructed images.…”

“…There are also many existing vision-based methods to reconstruct the 3D surface of a target organ while estimating the endoscope poses from a monocular endoscope video (see [8]- [10] for the surveys). The methods are ranging from shape-from-shading (SfS) [11]- [13], visual simultaneous localization and mapping (SLAM) [14]- [17], and structure-from-motion (SfM) [18]- [22]. However, most of existing works only have demonstrated the reconstruction result of a partial surface of the target organ, which is not sufficient for our localization purpose.…”

Real-Time Dense Reconstruction of Tissue Surface From Stereo Optical Video

“…EKF-SLAM suffers from low accuracy and is difficult for representing loop closing. Recently, Mahmoud et al [33] proposed a monocular vision-based dense tissue 3D reconstruction method by using ORB-SLAM [6] to estimate the camera motion. However, because ORB-SLAM is based on ORB features and RANSAC+P3P [34] for camera motion tracking and loop closing, its robustness is not satisfying with low texture scenes.…”

An augmented reality platform for interactive aerodynamic design and analysis

“…With this system, we do not expect to obtain a dense cloud of points, as it happens with other acquisition systems (such as RGB‐D devices or stereo systems), but to be able to take very precise measurements in real time with little computational cost. It is also important to point out that densification techniques can be applied a posteriori to obtain a denser 3D point cloud, such as the work of Mahmoud et al…”