Direct semi-dense SLAM for rolling shutter cameras

Abstract

In this paper, we present a monocular Direct and Semi-dense SLAM (Simultaneous Localization And Mapping) system for rolling shutter cameras. In a rolling shutter camera, the pose is different for each row of each image, and this yields poor pose estimates and poor structure estimates when using a state-of-the-art semi-dense direct method designed for global shutter cameras. To address this issue in tracking, we model the smooth and continuous camera trajectory using a B-spline curve of degree k−1 for poses in the Lie algebra, se(3). We solve for the camera poses at each row-time by a direct optimisation of photometric error as a function of the control points of the spline. Likewise for mapping, we develop generalised epipolar geometry for the rolling shutter case and solve for point depths using photometric error. Although each of these issues has been previously tackled, to the best of our knowledge ours is the first full solution to monocular, direct (feature-less) SLAM. We benchmark our method for pose accuracy and map accuracy against the state-of-the-art semi-dense SLAM system, LSD-SLAM, demonstrating the improved efficacy of our approach when using rolling shutter cameras via synthetic sequences with known ground-truth and real sequences.

Trajectory representation and landmark projection for continuous-time structure from motion

“…• We compare the split and SE(3) trajectory representations theoretically, and in a series of both synthetic and real data experiments. • We compare the performance and efficiency of three previously proposed ways to incorporate reprojection time into the optimization (Hedborg et al 2012;Furgale et al 2012;Lovegrove et al 2013;Kim et al 2016). • For completeness, we also describe our recently published spline error weighting approach to better balance the residuals in the optimization problem, and to automatically set the spline knot spacing based on desired trajectory accuracy (Ovrén and Forssén 2018).…”

Direct Sparse Odometry with Rolling Shutter

“…For direct rolling shutter approaches, real-time capability is a challenge. With our formulation, we are already much closer to real-time processing than the alternative approach in [12]. In future work, we will investigate further speedups of our implementation.…”

Rolling-Shutter Modelling for Direct Visual-Inertial Odometry

“…The RGB-D method in [1] uses not only a photometric error term, but also a geometric error term due to the availability of depth measurements. Direct monocular approaches have been presented in [22], [7]. While the first decouples motion and structure estimation and enforces the rolling-shutter constraint softly by introducing additional time variables, the latter performs a global bundle adjustment and explicitly solves the rolling-shutter constraint.…”