Where the action could be: Speakers look at graspable objects and meaningful scene regions when describing potential actions.

Abstract

The world is visually complex, yet we can efficiently describe it by extracting the information that is most relevant to convey. How do the properties of real-world scenes help us decide where to look and what to say? Image salience has been the dominant explanation for what drives visual attention and production as we describe displays, but new evidence shows scene meaning predicts attention better than image salience. Here we investigated the relevance of one aspect of meaning, graspability (the grasping interactions objects in the scene afford), given that affordances have been implicated in both visual and linguistic processing. We quantified image salience, meaning, and graspability for real-world scenes. In three eyetracking experiments, native English speakers described possible actions that could be carried out in a scene. We hypothesized that graspability would preferentially guide attention due to its task-relevance. In two experiments using stimuli from a previous study, meaning explained visual attention better than graspability or salience did, and graspability explained attention better than salience. In a third experiment we quantified image salience, meaning, graspability, and reach-weighted graspability for scenes that depicted reachable spaces containing graspable objects.Graspability and meaning explained attention equally well in the third experiment, and both explained attention better than salience. We conclude that speakers use object graspability to allocate attention to plan descriptions when scenes depict graspable objects within reach, and otherwise rely more on general meaning. The results shed light on what aspects of meaning guide attention during scene viewing in language production tasks.

Meaning and Expected Surfaces Combine to Guide Attention During Visual Search in Scenes

“…Spatial constraint interacts with image salience to guide attention during visual search (Ehinger et al, 2009;Torralba et al, 2006). Given the correlation between image salience and meaning in real-world scenes (Elazary & Itti, 2008;Henderson, 2003;Henderson et al, 2007;Henderson & Hayes, 2017Rehrig et al, 2020;Tatler et al, 2011) and the finding that meaning accounts for most if not all of the shared variance in predicting eye fixations when the intercorrelation between meaning and saliency is controlled (Hayes & Henderson, 2019;Henderson & Hayes, 2017Peacock et al, n.d., 2019bPeacock et al, n.d., , 2019aPeacock et al, n.d., , 2020Rehrig et al, 2020), spatial constraint might also interact with meaning to guide eye movements.…”

Towards an ecological-dynamics design framework for embodied-interaction conceptual learning: the case of dynamic mathematics environments

“…The idea of an affordance foregrounds the formative role of sensory perception as the human-environment functional bind. Perception is always perception-for-action (Rehrig et al 2020;Varela et al 1991). Thus, the interplay of perception and action is co-constructive and mutually serving, with perception guiding action, even as action promotes perceptual vantage (Fiebelkorn and Kastner 2019;Maturana and Varela 1992;Schroeder et al 2010).…”

Meaning Maps Capture the Density of Local Semantic Features in Scenes: A reply to Pedziwiatr, Kümmerer, Wallis, Bethge & Teufel (2021)

“…For example, we have generated what we call "contextualized" Meaning Maps by presenting exactly the same patches used in context-free Meaning Maps, but with each individual patch shown with its scene . We have also generated "Grasp Maps" using exactly the same patches with instructions focused on whether the region depicts an entity that can be grasped (Rehrig et al, 2020). Importantly, when the instructions are changed, subjects change their ratings to reflect the semantic features they are asked to rate, leading to different maps, even though the physical features are held constant.…”

“…And critically, it is a simple matter to extend the "classic" context-free Meaning Map approach to include additional types of semantic features, as we have done (Peacock et al, 2019;Rehrig et al, 2020). Importantly, whereas Meaning Maps can easily be extended to investigate a variety of semantic features, it is far less clear whether deep learning models like DG2 can ever in principle capture object-scene semantic features, or indeed any type of semantic feature.…”

“…PKWBT do a disservice to many fields of inquiry by incorrectly and unnecessarily dismissing a method that has already proven useful for investigating scene semantics across several domains of research, and that will likely continue to prove useful in new domains. For example, Meaning Maps have already been used to investigate scene memory (Bainbridge et al, 2019;Ramey et al, 2020), language production (Ferreira & Rehrig, 2019;Henderson et al, 2018;Rehrig et al, 2020), mind wandering (Krasich et al, 2020), active vision in immersive VR environments (Haskins et al, 2020), and infant attentional development (Klotz et al, 2021).…”