Information theoretic preattentive saliency: A closed-form solution

Abstract

Employing an information theoretic operational definition of bottom-up attention from the field of computational visual perception, a very general expression for saliency is provided. As opposed to many of the current approaches to determining a saliency map, there is no need for an explicit, data-driven density estimation. Given the features, descriptors, or filter bank that one wants to use to describe the image content at every position, we provide a closed-form expression for the associated saliency at that location. This, indeed, makes explicit that what is considered salient depends on how, i.e., by means of which features, image information is described. We illustrate our result by determining a few specific saliency maps based on particular choices of features. One of them makes the link with the mapping underlying well-known Harris interest points, which is a result recently obtained in isolation. Another choice of features is, rather loosely, inspired by the success of histogram of oriented gradient descriptors and proves to provide stateof-the-art results on a collaborative benchmark for region of interest detection.

Why Did This Model Forecast This Future? Closed-Form Temporal Saliency Towards Causal Explanations of Probabilistic Forecasts

“…Rather than defining saliency in a top-down manner as a function of some task-specific error metric, we have started from a more fundamental conception of bottom-up, or task-agnostic, saliency. Loog's [28]'s original definition pertains to preattentive saliency, which captures what is perceived to be subconsciously informative before conscious (attentive) processing by the brain. Here, a surprising or unexpected observation is salient.…”

“…In this work, our central idea for temporal saliency in forecasting settings is as follows: the saliency of a preceding observed window is related to the uncertainty over the future window of interest resulting from its observation. Our framework addresses all the aforementioned concerns we identified with saliency-based approaches in forecasting as follows: (i) the alternate 'causes' in our framework pertain to real preceding windows of actual observed behavior; (ii) our information-theoretic perspective builds upon a fundamental definition of saliency grounded in bottom-up preattentive human perception [28,29], and can be used to compute saliency for unseen test data where the ground-truth future is unavailable; (iii) we utilize a distribution over the futures that the model believes could have occurred over a future window rather than the single future that did occur: what a model considers plausible for a single forecast encapsulates the temporal patterns of behavior it has learned from the entire dataset; and (iv) we derive a closed-form expression for the saliency map that can be applied to any model that outputs a probability distribution for its forecasts without additional training or optimization.…”

“…Loog [28] developed a general closed-form expression for saliency using a surprisal-based operational definition of bottom-up attention from the field of computational visual perception. Let L : R n → R d be a general n-dimensional d-valued image.…”