Frameworks of analysis for the neural representation of animate objects and actions

Abstract

A variety of cell types exist in the temporal cortex providing high-level visual descriptions of bodies and their movements. We have investigated the sensitivity of such cells to different viewing conditions to determine the frame(s) of reference utilized in processing. The responses of the majority of cells in the upper bank of the superior temporal sulcus (areas TPO and PGa) found to be sensitive to static and dynamic information about the body were selective for one perspective view (e.g. right profile, reaching right or walking left). These cells can be considered to provide viewer-centred descriptions because they depend on the observer's vantage point. Viewer-centred descriptions could be used in guiding behaviour. They could also be used as an intermediate step for establishing view-independent responses of other cell types which responded to many or all perspective views selectively of the same object (e.g. head) or movement. These cells have the properties of object-centred descriptions, where the object viewed provides the frame of reference for describing the disposition of object parts and movements (e.g. head on top of shoulders, reaching across the body, walking forward ‘following the nose’). For some cells in the lower bank of the superior temporal sulcus (area TEa) the responses to body movements were related to the object or goal of the movements (e.g. reaching for or walking towards a specific place). This goal-centred sensitivity to interaction allowed the cells to be selectively activated in situations where human subjects would attribute causal and intentional relationships.

Understanding other people's actions: Intention and attention.

“…Neurons showing similar properties to those in area F5 have also been reported in monkeys within the superior temporal sulcus (STS) by Perrett and colleagues (Oram & Perrett, 1996;Perrett, Harries, Bevan, & Thomas, 1989;Perrett, Rolls, & Caan, 1982). For instance, in the lower bank of the STS, cells sensitive to perceived actions of the hand have been found.…”

“…In Experiment 1, more body information was available from the human model than from the robot model, and this additional information may have provided the cues necessary to influence observers' performance, even for the catch trials. In this respect, Perrett et al (1989) found effects due to sight of action when the entire body and face of the experimenter performing the action were visible. When only the hand of the experimenter was visible, cells were much less responsive.…”

Parietal maps of visual signals for bodily action planning

“…These studies have shown that specific observed-action exemplars depicted in brief videos can be decoded from the responses of AIP and phAIP neurons even across viewpoints (e.g., lateral/frontal), supporting the idea that neuronal populations in these areas encode OMA identity. In particular, we proposed (Orban et al 2021 ) that OMA identity is computed from two distinct STS signals reaching AIP, the first concerning body shape changes and originating in areas PGa/IPa (Vangeneugden et al 2009 ), and the second concerning the attainment of the goal by others’ hand–object interaction provided by area TEa (Perrett et al 1989 ). Therefore, we suggest that, as previously hypothesized for objects (Cisek 2007 ), the encoding of specific observed actions’ identity by AIP neurons endowed with motor properties can lead to the specification and selection of the potential motor actions required to interact with the observed agent, thereby extending the concept of affordances from objects to others’ actions.…”

Neural Architectures for Robot Intelligence

“…Since hands are a major focus of action, we -but also other higher primates -are very good at the visual recognition of hands, their motions as well as their highly variable postures and -at an even higher level -their actions. Most interestingly, such recognition tasks were found to be correlated with the activity of neurons (located in the superior temporal sulcus) that show rather selective responses to visually perceived hand actions [35], see Fig. 2.…”

“…Figure 2: Selective neuron response to visually perceived hand actions (from [35]). The bottom row shows the associated activity trace of a neuron (the same for all trials) in the STS (superior temporal sulcus) of a macaque when the animal observes the action sequence (1 s duration, indicated by calibration bar at the bottom) depicted in the corresponding column above.…”