Actionable Information in Vision

Abstract

Summary. A notion of visual information is introduced as the complexity not of the raw images, but of the images after the effects of nuisance factors such as viewpoint and illumination are discounted. It is rooted in ideas of J. J. Gibson, and stands in contrast to traditional information as entropy or coding length of the data regardless of its use, and regardless of the nuisance factors affecting it. The non-invertibility of nuisances such as occlusion and quantization induces an "information gap" that can only be bridged by controlling the data acquisition process. Measuring visual information entails early vision operations, tailored to the structure of the nuisances so as to be "lossless" with respect to visual decision and control tasks (as opposed to data transmission and storage tasks implicit in communications theory). The definition of visual information suggests desirable properties that a visual representation should possess to best accomplish vision-based decision and control tasks.

Visual-inertial navigation, mapping and localization: A scalable real-time causal approach

“…Already the image, in the local frame, is by construction invariant to similarity transformations. To achieve invariance to contrast, we replace the image with the gradient direction at each point, since the gradient direction is dual to the geometry of the level lines which is a maximal contrast invariant statistic (Soatto 2009). However, instead of coarsely binning the descriptor to achieve some kind of insensitivity to viewpoint changes beyond similarities, as in SIFT and HOG (Dalal and Triggs 2005), we have the luxury of tracking, which gives us samples of the image in the local frame Fig.…”

Exploiting Photometric Information for Planning Under Uncertainty

“…Optimal Rapidly-exploring Random Trees (RRT*s) [10] have been widely used in path planning problems and their extension to Rapidly-exploring Random Belief Trees (RRBTs) [7] takes pose uncertainty into account and avoids collisions. Selecting sequences of viewpoints that optimize for a certain task (e.g, pose estimation or map uncertainty minimization) is referred to as active perception [11,12]. While previous papers on active perception relied on using range sensors (e.g, [8]), Davison and Murray [13] were among the first to use vision sensors (a stereo camera setup) to select where the camera should look to reduce the pose drift during visual SLAM.…”

“…The fundamental steps of the perception-aware RRT* are summarized in Algorithm 1. At each iteration, it samples a new state from the state space and connects it to the nearest vertex (lines [3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19]. Next, the function Near() checks on the vertices within a ball, centered at the sampled state (see [10]), and propagate the pose covariance from these vertices to the newly sampled one.…”

Dynamic Vision for Control

“…Thus this "control-authority/actionable information" tradeo↵ extends "rate/distortion" theory when the underlying task is not the storage or transmission of data, but its use in decision and control tasks. This construction is described in [22].…”