Neural Representations of Observed Actions Generalize across Static and Dynamic Visual Input

Abstract

People interact with entities in the environment in distinct and categorizable ways (e.g., is). We can recognize these action categories across variations in actors, objects, and settings; moreover, we can recognize them from both dynamic and static visual input. However, the neural systems that support action recognition across these perceptual differences are unclear. Here, we used multivoxel pattern analysis of fMRI data to identify brain regions that support visual action categorization in a format-independent way. Human participants were scanned while viewing eight categories of interactions (e.g., ) depicted in two visual formats: (1) visually controlled videos of two interacting actors and (2) visually varied photographs selected from the internet involving different actors, objects, and settings. Action category was decodable across visual formats in bilateral inferior parietal, bilateral occipitotemporal, left premotor, and left middle frontal cortex. In most of these regions, the representational similarity of action categories was consistent across subjects and visual formats, a property that can contribute to a common understanding of actions among individuals. These results suggest that the identified brain regions support action category codes that are important for action recognition and action understanding. Humans tend to interpret the observed actions of others in terms of categories that are invariant to incidental features: whether a girl pushes a boy or a button and whether we see it in real-time or in a single snapshot, it is still Here, we investigated the brain systems that facilitate the visual recognition of these action categories across such differences. Using fMRI, we identified several areas of parietal, occipitotemporal, and frontal cortex that exhibit action category codes that are similar across viewing of dynamic videos and still photographs. Our results provide strong evidence for the involvement of these brain regions in recognizing the way that people interact physically with objects and other people.

Sociality and Interaction Envelope Organize Visual Action Representations

“…However, common verbs are not always commonly-seen actions (e.g., "can"), and many commonly-seen actions are not often talked about (e.g., "chopping vegetables"). Further, using verbs (particularly without considering the type of verb) can unintentionally constrain one's hypothesis space by implying that actions that look very different but are described by the same verb ("pushing a button" vs. "pushing a person") share a neural representation (Hafri et al, 2017). To understand action observation at a more perceptual, rather than conceptual level, we instead sampled our action stimuli based on common human experiences, using the American Time Use Survey as a guide (see Methods; cf.…”

“…Therefore, it is critical to consider more widespread action responses across broad swathes of occipitotemporal and parietal cortices (e.g. Hafri et al, 2017;Wurm & Caramazza, 2017). Given this, we used a novel method to identify voxels that reliably differentiate among different actions (Tarhan & Konkle, under review), rather than constraining our analyses to specific regions of interest emphasized in the literature.…”

Dissecting the visual perception of body shape with the Garner selective attention paradigm

“…static/dynamic; verbal/visual) spans the lateral and ventral occipitotemporal cortex (e.g. Hafri et al, 2017;O'Toole et al, 2014;Wurm & Lingnau, 2015; for review see Lingnau & Downing, 2015).…”

Distinct profiles of temporal and frontoparietal cortex in representing actions across vision and language

“…This conception has recently been challenged by demonstrating that posterior temporal cortex encodes action representations (e.g. of opening and closing) that generalize across a range of perceptual features, such as the body parts (Vannuscorps et al, 2018) and movements used to carry out an action (Wurm and Lingnau, 2015;Vannuscorps et al, 2018), the type of object involved in an action (Wurm and Lingnau, 2015;Vannuscorps et al, 2018), and whether an action is recognized from photographs or videos (Hafri et al, 2017). These findings suggest that temporal cortex encodes action representations that abstract away from various details of a perceived action.…”