Where the action could be: Speakers look at graspable objects and meaningful scene regions when describing potential actions

Abstract

The world is visually complex, yet we can efficiently describe it by extracting the information that is most relevant to convey. How do the properties of real-world scenes help us decide where to look and what to say? Image salience has been the dominant explanation for what drives visual attention and production as we describe displays, but new evidence shows scene meaning predicts attention better than image salience. Here we investigated the relevance of one aspect of meaning, graspability (the grasping interactions objects in the scene afford), given that affordances have been implicated in both visual and linguistic processing. We quantified image salience, meaning, and graspability for real-world scenes. In three eyetracking experiments, native English speakers described possible actions that could be carried out in a scene. We hypothesized that graspability would preferentially guide attention due to its task-relevance. In two experiments using stimuli from a previous study, meaning explained visual attention better than graspability or salience did, and graspability explained attention better than salience. In a third experiment we quantified image salience, meaning, graspability, and reach-weighted graspability for scenes that depicted reachable spaces containing graspable objects. Graspability and meaning explained attention equally well in the third experiment, and both explained attention better than salience. We conclude that speakers use object graspability to allocate attention to plan descriptions when scenes depict graspable objects within reach, and otherwise rely more on general meaning. The results shed light on what aspects of meaning guide attention during scene viewing in language production tasks.

Rapid Extraction of the Spatial Distribution of Physical Saliency and Semantic Informativeness from Natural Scenes in the Human Brain

“…The rapid extraction of meaning-related information is consistent with behavioral work showing that the spatial distribution of meaningful scene features exerts an influence on even the initial shift of overt attention in real-world scenes Hayes, 2017, 2018;Hayes and Henderson, 2019;Peacock et al, 2020;Rehrig et al, 2020). Specifically, the 86 ms onset latency of the meaning-related activity observed in the present study is sufficiently fast to potentially influence even the earliest shifts of overt attention (Thorpe et al, 1996;Fabre-Thorpe et al, 2001;Gordon, 2004).…”

“…Several recent studies have shown that eye movement patterns are predicted better by meaning maps than by physical saliency (Henderson and Hayes, 2017;Henderson et al, 2019). This advantage has been observed across multiple tasks, including visual search (Hayes and Henderson, 2019), simple free viewing (Peacock et al, 2019a), and scene and action description (Henderson and Hayes, 2018;Rehrig et al, 2020). The predictive advantage of meaning maps is present even when the task is to count the number of physically salient scene regions (Peacock et al, 2019b).…”

Center Bias Does Not Account for the Advantage of Meaning Over Salience in Attentional Guidance During Scene Viewing

“…Overall, the results are consistent with previous meaning mapping work using a traditional central fixation start location Hayes, 2017, 2018;Peacock et al, 2019a,b;Rehrig et al, 2020) in which we found that early eye movements were more related to meaning than saliency. The present findings verify that the advantage of meaning over salience observed by previous meaning mapping studies was not simply due to an advantage for meaning at scene centers induced by the use of an initial central fixation location.…”

“…Recent work in real-world attentional guidance has shown that meaning maps representing the semantic features of local scene regions are more highly related to fixation distributions than are saliency maps representing image feature differences, a result that has been replicated across a number of viewing tasks Hayes, 2017, 2018;Hayes and Henderson, 2019b;Peacock et al, 2019a,b;Rehrig et al, 2020). However, centers of photographs may contain greater meaningful information and image features than in scene peripheries, and for that reason participants might strategically fixate centrally (Parkhurst et al, 2002;Tatler, 2007;Tseng et al, 2009;Bindemann, 2010;Rothkegal et al, 2017;van Renswoude et al, 2019), conflating whether meaning actually guides attention better than image salience or whether this phenomenon is due to central fixation bias.…”

“…In studies directly comparing meaning maps and saliency maps, meaning has been found to be a significantly better predictor of visual attention than image salience. This advantage for meaning over salience was observed across viewing tasks such as aesthetic judgment and memorization ( Henderson and Hayes, 2017 , 2018 ), scene description and action description ( Henderson et al, 2018 ; Rehrig et al, 2020 ), and visual search ( Hayes and Henderson, 2019b ). These results have also been obtained using viewing tasks that do not require semantic analysis of the scene, such as counting bright, physically salient scene regions ( Peacock et al, 2019a ), visual search for arbitrarily placed letter targets ( Hayes and Henderson, 2019b ), and free viewing ( Peacock et al, 2019b ).…”