How What We See and What We Know Influence Iconic Gesture Production

Abstract

In face-to-face communication, speakers typically integrate information acquired through different sources, including what they see and what they know, into their communicative messages. In this study, we asked how these different input sources influence the frequency and type of iconic gestures produced by speakers during a communication task, under two degrees of task complexity. Specifically, we investigated whether speakers gestured differently when they had to describe an object presented to them as an image or as a written word (input modality) and, additionally, when they were allowed to explicitly name the object or not (task complexity). Our results show that speakers produced more gestures when they attended to a picture. Further, speakers more often gesturally depicted shape information when attended to an image, and they demonstrated the function of an object more often when they attended to a word. However, when we increased the complexity of the task by forbidding speakers to name the target objects, these patterns disappeared, suggesting that speakers may have strategically adapted their use of iconic strategies to better meet the task’s goals. Our study also revealed (independent) effects of object manipulability on the type of gestures produced by speakers and, in general, it highlighted a predominance of molding and handling gestures. These gestures may reflect stronger motoric and haptic simulations, lending support to activation-based gesture production accounts.

Gesture as simulated action: Revisiting the framework

“…It seems likely the spatial characteristics of the objects were particularly salient after seeing the objects depicted. In contrast, Masson-Carro et al (2017) found that participants were more likely to produce gestures that mimicked holding or interacting with objects when they described objects that are readily manipulable (e.g., a tennis racket), and they produced such gestures most often when they had read the name of the object rather than seen it depicted. It seems likely that manipulable objects activate motor imagery more strongly than spatial imagery, particularly when the objects' spatial characteristics have not just been viewed.…”

“…Although speculative, there is some empirical support for this explanation that the effect of stimulus presentation modality can depend on aspects of the communicative situation. Masson-Carro, Goudbeek, and Krahmer (2017) found that participants gestured more when describing an object to someone else when the object was indicated with a picture than when it was indicated with a verbal label. The effect disappeared, however, when the task was made more difficult by preventing participants from naming the object, perhaps because participants adopted a very different strategy when not allowed to use the name.…”

“…Perhaps it is simply more difficult to describe information that was presented as images than to describe the same information when it was presented as words, and this increased difficulty leads to more gestures. Certainly, prohibiting speakers from naming the object they are describing (as in Masson-Carro et al, 2017) imposes an additional cognitive load that could result in more gestures.…”

“…For example, when speakers form a spatial image of an object that emphasizes how the parts of the object are arranged in space, they should be particularly likely to produce gestures that move between the various parts of the shape. Indeed, Masson-Carro et al (2017) found that such tracing or molding gestures were particularly likely to occur when speakers were describing nonmanipulable objects (e.g., a traffic light), particularly when they had seen a picture (rather than read the name) of the object. It seems likely the spatial characteristics of the objects were particularly salient after seeing the objects depicted.…”

Actions as a Basis for Online Embodied Concepts

“…Additionally, when reading a word or seeing an image of an object, the different types of information inform different forms of gestural expression. Researchers have demonstrated that seeing an image of an object leads to more gesture specifically depicting the physical characteristics of the object, whereas reading the name of an object is correlated with relatively less gesture production but the produced gestures are focused on the utility of the object (Masson-Carro et al 2017). Furthermore, objects with high affordances are associated with more representational gestures and with gestures based on how the object is manipulated in the real world (Masson-Carro et al 2016).…”