On the set of images modulo viewpoint and contrast changes

Abstract

We consider regions of images that exhibit smooth statistics, and pose the question of characterizing the "essence" of these regions that matters for recognition. Ideally, this would be a statistic (a function of the image) that does not depend on viewpoint and illumination, and yet is sufficient for the task. In this manuscript, we show that such statistics exist. That is, one can compute deterministic functions of the image that contain all the "information" present in the original image, except for the effects of viewpoint and illumination. We also show that such statistics are supported on a "thin" (zero-measure) subset of the image domain, and thus the "information" in an image that is relevant for recognition is sparse. Yet, from this thin set one can reconstruct an image that is equivalent to the original up to a change of viewpoint and local illumination (contrast). Finally, we formalize the notion of "information" an image contains for the purpose of viewpoint-and illuminationinvariant tasks, which we call "actionable information" following ideas of J. J. Gibson.

Actionable Information in Vision

“…Viewpoint g and contrast h act on the image as groups, in the absence of occlusions and cast shadows, and therefore can be inverted [72]. In other words, the effects of a viewpoint and contrast change, away from visibility artifacts, can be "neutralized" in a single image, and an invariant sufficient statistic can, at least in principle, be computed [72].…”

“…In other words, the effects of a viewpoint and contrast change, away from visibility artifacts, can be "neutralized" in a single image, and an invariant sufficient statistic can, at least in principle, be computed [72]. Note that the notion of sufficient statistic in this case is with respect to any distribution, since it is possible to reconstruct an individual realization of the scene regardless of the nuisance.…”

“…It may be puzzling that the statistics that are invariant to contrast (the geometry of the iso-contours of the image [19]) are not invariant to viewpoint, and those that are invariant to viewpoint (the intensity of the image warped onto a canonical domain [78]) are not invariant to contrast. The conundrum was recently solved in [72] where it was shown that the Attributed Reeb Tree (ART) of a (portion of an) image is the viewpoint-contrast invariant sufficient statistic. The ART stores the label (maximum, minimum, saddle), relative ordering and connectivity of extrema of the function f in (1.3) and is supported on a zero-measure subset of the image domain, so it is somewhat surprising that this "thin" object is actually equivalent to the entire image but for the effects of viewpoint and contrast changes.…”

“…Without loss of generality [72], I represent changes of viewpoint with diffeomorphic domain deformations w : D ⊂ R 2 → R 2 . This model is viable only away from visibility artifacts (occlusions, cast shadows), which are discussed in Sect.…”

Modeling Shapes with Higher-Order Graphs: Methodology and Applications

“…Regarding the problem complexity and the algorithm design, we can see from the above discussion that such extrinsic factors are a main source of shape variability [36], the removal of which will largely reduce the complexity of shape matching and inference. The problem can become much easier if we only need to deal with the intrinsic shape variability, which refers to the residual (e.g., intra-class variability, noise) after ruling out the effect of extrinsic factors.…”