Tightly Integrated Sensor Fusion for Robust Visual Tracking

Abstract

This paper presents novel methods for increasing the robustness of visual tracking systems by incorporating information from inertial sensors. We show that more can be achieved than simply combining the sensor data within a statistical filter. In particular we show how, in addition to using inertial data to provide predictions for the visual sensor, this data can also be used to provide an estimate of motion blur for each feature and this can be used to dynamically tune the parameters of each feature detector in the visual sensor. This allows the system to obtain useful information from the visual sensor even in the presence of substantial motion blur. Finally, the visual sensor can be used to calibrate the parameters of the inertial sensor to eliminate drift.

Improving the Agility of Keyframe-Based SLAM

“…For a start, they are one-dimensional, and so locally is motion blur: One would therefore expect that even in a heavily blurred image, some edge features (those parallel to the local direction of blur) may remain intact. Further, even edges which are affected by blur can be tracked: Klein and Drummond [8] demonstrated that blurred edges can be used for tracking in real-time if an estimate for the magnitude of motion blur is known a priori. This motivates us to include intensity edges into the SLAM map.…”

“…Rapid unmodelled accelerations cause problems for data association and can further lead to the use of an incorrect blur estimate for edge tracking -for example, when the camera rotates suddenly from rest. In [8] Klein and Drummond employed inertial sensors to provide a rotation prediction for each frame to work around this problem. Here we propose an alternative: In Section 6 we describe a procedure to estimate inter-frame rotation by full-frame direct minimisation.…”

“…This is equivalent to projecting the rotated edgelet's z-axis into the image as done in [10]. Once projected, edgelets are measured directly in the full-size image, in a manner similar to most edge trackers [17,8,9]. The nearest locally-straight image edge is found by placing an edge of length 18 pixels in the image at the hypothesised position, populating five sample points along this edge, and then performing perpendicular searches at each sample point to find local gradient maxima of the correct polarity to sub-pixel accuracy.…”

“…Each edgelet is measured by performing a single perpendicular search from the edgelet center, providing a perpendicular distance measure ǫ. To provide some resilience to motion blur, we adopt the search method of Klein and Drummond [8]: Instead of searching for an intensity step in the image, the motion blur expected for each edgelet is calculated from a motion model, and the image is searched for an intensity ramp of the same length as the expected blur. The ramp search is performed by extracting a 1D intensity signal sampled along the edge normal, and cross-correlating with a single cycle of a sawtooth wave of period l. l is the edge-normal component of expected motion blur and is calculated by simply multiplying the expected camera motion γ C during exposure by each edgelet's measurement Jacobian:…”

Towards Coordinated Precision Assembly with Robot Teams

“…In addition, visual perception is highly challenged in many cases: occlusions, cluttered backgrounds, and image blurring because of fast motions either in objects or camera. To overcome these limitations of visual perception, it is often combined with motion estimation [13] or tactile sensing [1,10]. Skotheim et al [22] manipulation.…”