Robust Direct Visual Localisation using Normalised Information Distance

Abstract

Real-time visual localisation is a key technology enabling mobile location applications [7], virtual and augmented reality [1] and robotics [3]. The recent availablity of low-cost GPU hardware and GPGPU programming has enabled a new class of 'direct' visual localisation methods that make use of every pixel from an input image for tracking and matching [6], in contrast to traditional feature-based methods that only use a subset of the input image. The additional information available to direct methods localising against a dense 3D map increases robustness against typical failure modes for feature-based methods, such as motion blur and viewpoint change [2]. In this paper we present a visual localisation system which utilises the entropy-based Normalised Information Distance (NID) metric for image registration.For computational reasons, existing direct methods generally minimise a cost function based on photometric error on a per-pixel basis (e.g. [6]), which assumes both the live image and the reference map are embedded in the same space. Equation (1) defines such a metric based on the sum of squared differences between corresponding pixels in a reference image (I r ) and a synthetic image (I s ).whereT is a pixel location within the image.Although photoconsistency is efficient to compute and find derivatives for (in order to use in an optimisation framework), as mentioned in [6] it suffers from a number of limitations. Principally, it requires I s provide a photorealistic rendering of the scene, such that the resulting synthetic image matches the reference image I r on a pixel-by-pixel basis. A true match under significant appearance changes would require modelling of the material and illumination properties of the real-world environment, along with the response of the sensor. This limitation restricts photoconsistency to applications involving frame-to-frame tracking, where the synthetic image I s can be derived from a warping of the previous reference image I r , or applications in small indoor environments with controlled illumination where the scene does not change over time [6].In this paper we instead make use of the Normalised Information Distance (NID) metric, given by Equation (2) [4].where H(I r , I s ) and MI(I r , I s ) are the joint entropy and mutual information of the two images respectively, defined as follows:where H (I r ) is defined similarly to H (I s ). p s and p r,s are the marginal and joint discrete distributions of the images I r and I s , represented by nbin discrete histograms where a and b are individual bin indices.As NID is not a function of the actual values of the pixels in the image, but of their distribution, NID provides robustness to illumination change and sensor modality. Unlike mutual information, NID is a true metric, which satisfies the triangle inequality, and is normalised over the number of pixels used in the calculation, thus allowing comparisons between image pairs with differing amounts of overlap.In order to use NID as a cost function in a gradient-based optimisatio...

A survey on Visual-Based Localization: On the benefit of heterogeneous data

“…Consistent 3D models are also used to perform VBL task. For indoor localization, works from [183,145] use textured model reconstructed from RGB-D sensor [183] or hand-crafted with dedicated software [145]. City-scale models are used by [10,157,146,144,29] to perform outdoor VBL.…”

“…Another widely used method for combining data of different types consists of projecting one of the engaged data into the representation space of the other. For instance, lot of methods consider the challenging problem of registering photographies upon 3D models [180,80,7,145,146,144]. Similarity comparison is performed thanks to synthetic images generated from the 3D models [170,117,10,157].…”

Laser Map Aided Visual Inertial Localization in Changing Environment

“…Since visual and laser information are represented in different modalities, there are generally two ways to align vision-tracked trajectories with the laser map [7]. One way [6]- [8] is synthesizing 2D images from the 3D laser map using the intensity or depth information. The currently observed images are matched with the synthesized ones to calculate the relative transformations.…”

Fast Monte-Carlo Localization on Aerial Vehicles Using Approximate Continuous Belief Representations

“…Prior works exploiting known map appearance for precise monocular pose estimation [10,17,19,22] employ a textured depth map within an iterative optimization framework to compute the warp that minimizes a photometric cost function between a rendered image and the live image such as the Normalized Information Distance [19], that is robust to illumination change, or a Sum of Squared Differences cost function with an affine illumination model to tackle illumination change [17]. Both algorithms rely on initialization for tracking via a GPS prior or an ORB-based bagof-words approach, respectively, and expensive ray casted dense textured data for refinement.…”