A proto-object-based computational model for visual saliency

Abstract

State-of-the-art bottom-up saliency models often assign high saliency values at or near high-contrast edges, whereas people tend to look within the regions delineated by those edges, namely the objects. To resolve this inconsistency, in this work we estimate saliency at the level of coherent image regions. According to object-based attention theory, the human brain groups similar pixels into coherent regions, which are called proto-objects. The saliency of these proto-objects is estimated and incorporated together. As usual, attention is given to the most salient image regions. In this paper we employ state-of-the-art computer vision techniques to implement a proto-object-based model for visual attention. Particularly, a hierarchical image segmentation algorithm is used to extract proto-objects. The two most powerful ways to estimate saliency, rarity-based and contrast-based saliency, are generalized to assess the saliency at the proto-object level. The rarity-based saliency assesses if the proto-object contains rare or outstanding details. The contrast-based saliency estimates how much the proto-object differs from the surroundings. However, not all image regions with high contrast to the surroundings attract human attention. We take this into account by distinguishing between external and internal contrast-based saliency. Where the external contrast-based saliency estimates the difference between the proto-object and the rest of the image, the internal contrast-based saliency estimates the complexity of the proto-object itself. We evaluate the performance of the proposed method and its components on two challenging eye-fixation datasets (Judd, Ehinger, Durand, & Torralba, 2009; Subramanian, Katti, Sebe, Kankanhalli, & Chua, 2010). The results show the importance of rarity-based and both external and internal contrast-based saliency in fixation prediction. Moreover, the comparison with state-of-the-art computational models for visual saliency demonstrates the advantage of proto-objects as units of analysis.

Rapid scene perception with tragic consequences: observers miss perceiving vulnerable road users, especially in crowded traffic scenes

“…Scene processing begins with a massive array of visual information that is sampled during eye fixations. The array is subdivided and grouped into potential objects and background structures during early and intermediate visual processing (e.g., Itti & Koch, 2000;Xu & Chun, 2009;Yanulevskaya et al, 2013). Determining object and region boundaries is complex, because stimulus information is locally ambiguous, as has been noted (e.g., Granlund, 1999;Tsotsos, 1990Tsotsos, , 2001.…”

“…Conscious identification of an object requires individuation (e.g., Xu & Chun, 2009). Different regions of the processed scene compete with each other for individuation, and the likelihood that a region will win (be individuated and represented) increases with stimulus factors such as the region's size and contrast from the background (e.g., Itti & Koch, 2000;Yanulevskaya et al, 2013). In regions away from the center of fixation (parafoveal and peripheral regions), the competition for individuation becomes especially high, because neural resources are increasingly scarce; as a result, objects crowd other objects and make them consciously imperceptible (e.g., Whitney & Levi, 2011).…”

“…The consequence is significant perceptual limitations and, in particular, a limitation on perceiving multiple objects during the early stages of scene perception (e.g., Franconeri et al, 2013;Sanocki, 1991;Xu & Chun, 2009). Thus, early vision can be viewed as a competition between candidate regions for individuation as an object (e.g., Itti & Koch, 2000;Xu & Chun, 2009;Yanulevskaya, Uijlings, Geusebroek, Sebe, & Smeulders, 2013). As we elaborate below, larger and more salient regions have an advantage in this competition, because they activate larger neural populations.…”

Gaze-to-object mapping during visual search in 3D virtual environments

“…The ultimate goal is, however, to utilize GTOM approaches to non-artificial stimuli or stimuli where an object segmentation is simply not available. For these applications, gaze processing has to be combined with vision algorithms (e.g., proto-object maps [Yanulevskaya et al 2013]) that can automatically segment parts in an image which a user perceives as objects.…”

Object-level saliency detection with color attributes

“…To link objectness with saliency, local saliencies were computed in randomly sampling a large number of windows [27,28]. Recent study [29] suggested that the unit of attention depends on the task, the field of view and the observer's intention. Object proposal generation and salient object detection are tightly linked among human subjects [21].…”