Can you handle this? The impact of object affordances on how co-speech gestures are produced

Abstract

Hand gestures are tightly coupled with speech and with action. Hence, recent accounts have emphasised the idea that simulations of spatio-motoric imagery underlie the production of co-speech gestures. In this study, we suggest that action simulations directly influence the iconic strategies used by speakers to translate aspects of their mental representations into gesture. Using a classic referential paradigm, we investigate how speakers respond gesturally to the affordances of objects, by comparing the effects of describing objects that afford action performance (such as tools) and those that do not, on gesture production. Our results suggest that affordances play a key role in determining the amount of representational (but not non-representational) gestures produced by speakers, and the techniques chosen to depict such objects. To our knowledge, this is the first study to systematically show a connection between object characteristics and representation techniques in spontaneous gesture production during the depiction of static referents.

How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis.

“…This pattern holds, even when controlling for the objects' spatiality, concreteness, and ability for selfproduced movement (Hostetter, 2014). Moreover, when people describe manipulable objects, they tend to produce gestures that depict the physical actions involved in using or handling those objects (Masson-Carro et al, 2015).…”

“…Indeed, people produce more co-speech gestures when they talk about motor imagery (e.g., explain how to wrap a present) than when they talk about visual imagery (e.g., describe your favorite painting) or about abstract information (e.g., express your view on the use of a single currency in Europe) (Feyereisen & Havard, 1999). Furthermore, in object description tasks, people produce more co-speech gestures when they describe objects that are highly manipulable, such as a stapler, than when they describe objects that people do not typically manipulate with their hands, such as a fence (Hostetter, 2014;Masson-Carro et al, 2015;Pine, Gurney & Fletcher, 2010). This pattern holds, even when controlling for the objects' spatiality, concreteness, and ability for selfproduced movement (Hostetter, 2014).…”

“…First, people produce co-speech gestures more frequently when speaking is more challenging, and they produce co-thought gestures more frequently when problem solving is more challenging (e.g., for co-speech gestures: Kita & Davies, 2009, Rauscher et al, 1996Wesp et al, 2001; for cothought gestures: Chu & Kita, 2011). Second, people produce both co-speech and cothought gestures more frequently when talking or thinking about stimulus objects that afford action more strongly, compared to talking or thinking about objects that afford action less strongly (Chu & Kita, 2015;Hostetter, 2014;Masson-Carro et al, 2015;Pine, Gurney & Fletcher, 2010). Third, there are parallel changes in co-speech and co-thought gestures over the course of learning to solve problems.…”

“…When complex information (e.g., the shape of a vase) is gesturally expressed, a single gesture may not be able to express all relevant aspects of the information, and each gesture may then focus on a particular aspect (e.g., the shape of the opening, the contour outline from a particular viewpoint). What is expressed by a gesture may be determined by affordances (Gibson, 1979) of the referent (Chu & Kita, 2015;Masson-Carro, Goudbeek & Krahmer, 2015) or by other top-down factors (e.g., an…”

“…For example, gestures may schematize the movement of an object in three ways: (1) tracing the trajectory with a point, (2) by moving the hand along the trajectory with a handshape that represents some aspects of the object or the surface (which supports the object), (3) the hand depicting grasping of the object (Chu & Kita, 2008;Sekine, Wood & Kita, under review). Gestures may schematize an object also in three ways: (1) tracing its outline, (2) moving the hands as if to touch (sculpt) the object surfaces, (3) using handshape to represent the shape (e.g., a flat hand to represent a flat object) (Masson-Carro et al, 2015;Müller, 1998). Focusing only on the trajectory of object movement and abstracting away the object shape may be useful when describing rotation of the object, as in Figure 3.…”