Coordinating Gesture, Word, and Diagram: Explanations for Experts and Novices

Abstract

Successful explanations are a symphony of gesture, language, and props. Here, we show how they are orchestrated in an experiment in which students explained complex systems to imagined novices and experts. Visual-spatial communication--diagram and gesture--was key; it represents thought more directly than language. The props, real or virtual diagrams created from gestures, served as the stage for explanations, enriched by language and enlivened by deictic gestures to convey structure and iconic gestures to enact the behavior and functionality of the systems.Explanations to novices packed in more information than explanations to experts, emphasizing the information about action that is difficult for novices, and expressing information in multiple ways, using both virtual models created by gestures and visible ones. g., Clark, 1996;Engle, 1998;Goldin-Meadow, 2003;McNeill, 1992;Tversky, Heiser, Lee, & Daniel, 2009). Both gestures and sketches are spatial forms of communication; they use elements or actions in space and the spatial relations among them to represent meanings that can be visuospatial, as in maps or architectural plans, or abstract, as in schedules, corporate charts, decision trees, and scientific diagrams. One advantage of visuo-spatial forms of communication is that they apply people's highly-practiced spatial skills to making abstract inferences. Another advantage of forms of visuo-spatial communication, gestures, sketches, and props, is that they can represent content more directly than words, using elements, relations, and actions in space. KeywordsThey can show structure, that is, relative locations of people or parts, and they can demonstrate actions (e. g., Tversky, 2011; Tversky and Kessell, in press). Together, word, gesture, sketch, and prop can create and annotate models that represent the situation speakers are trying to convey (e. g., Enfield, 2003;Engle, 1998;Emmorey, Tversky, & Taylor, 2000). These multimodal models draw from speakers' mental models and can serve to create mental models in the minds of their interlocutors. These layers of communication, word, gesture, props (and more) complement and supplement one another, and are coordinated to create meaning (e. g., Clark, 1996;Goldin-Meadow, 2003;McNeill, 1992). Here, we explore how gesture, word, and diagram are used and coordinated in explanations of complex scientific processes to novices and to experts. EXPLANATIONS FOR EXPERTS AND NOVICES 4Countless human interactions include explanations to novices or to experts. Everyday conversation typically entails both: when I relate something to you that I know but you do not, and when you check your understanding by explaining it back to me. We are all experts in our own experiences, and our audiences are necessarily novices. Collaborations provide many more examples. They usually consist of individuals with different expertise, so that they often entail many explanations to experts and to novices. Educational settings also provide such situations.Explaining to a student is expla...

From hands to minds: Gestures promote understanding

“…Integrated sequences of gestures can create virtual models of complex spaces or complex sequences of actions (Emmorey et al, 2000; Heiser et al, 2004; Kang et al, 2014; Kirsh, 2013; Tversky, Heiser, Lee, & Daniel, 2009; Jamalian et al, 2013). Here, we transferred gestures for expression to gestures for teaching and learning; we found that an integrated series of gestures congruent with action can deepen understanding of the actions of a dynamic system.…”

“…There are parallel cases for gesture; that is, the sequence of gestures form the core of the communication, and the words serve to annotate the gestures (e.g., Emmorey et al, 2000; Kirsh, 2013). In many instances, the three—gesture, talk, and diagram—work together, complementing and supplementing each other (e.g., Engle, 1998; Kang et al, 2014; Heiser et al, 2004). Examples abound, for example, in sports, dance, musical instruments, cooking, and more.…”

“…Gestures are actions; they should be natural for conveying actions (e.g., Cartmill, Beilock, & Goldin-Meadow, 2012; Hostetter & Alibali, 2008). Numerous studies have shown that people spontaneously gesture when explaining to themselves or to others (e.g., Alibali, Bassok, Solomon, Syc, & Goldin-Meadow, 1999; Alibali, Spencer, Knox, & Kita, 2011; Atit, Gagnier, & Shipley, 2014; Cartmill et al, 2012; Chu & Kita, 2011; Emmorey, Tversky, & Taylor, 2000; Ehrlich, Levine, & Goldin-Meadow, 2006; Engle, 1998; Goldin-Meadow & Beilock, 2010; Goldin-Meadow & Alibali, 1999; Goldin-Meadow, Kim, & Singer, 1999; Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001; Gukson, Goldin-Meadow, Newcombe, & Shipley, 2013; Hostetter & Alibali, 2008; Kang, Tversky, & Black, 2014; Schwartz & Black, 1996). In many cases, gestures carry information that is not carried in speech.…”

“…Gestures are frequently produced spontaneously to express both structure and action (e.g., Atit et al, 2014; Cartmill et al, 2012; Chu & Kita, 2011; Emmorey et al, 2000; Enfield, 2003; Engle, 1998; Goldin-Meadow & Beilock, 2010; Gukson et al, 2013; Kang et al, 2014). In previous research showing effects of communicative gestures that convey actions, the gestures used have been single actions on visible objects, such as lifting a disk (Goldin-Meadow & Beilock, 2010), counting (Carlson, Avraamides, Cary, & Strasberg, 2007) or rotating an imagined object (Alibali et al, 2011; Chu & Kita, 2011; Schwartz & Black, 1996).…”

Spatial analogies pervade complex relational reasoning: Evidence from spontaneous gestures

“…A major finding of this line of work is that people often develop mental models of the system that are visuospatial in nature (Gentner & Stevens, 1983; Hegarty, 2004). One line of evidence for the visuospatial character of these models is that, when reasoning about or explaining such systems, people often produce diagrams (Forbus, Usher, Lovett, & Wetzel, 2011; Novick, 2001; Tversky, 2011) or gestures (e.g., Kang, Tversky, & Black, 2014; Nathan & Martinez, 2015; Schwartz & Black, 1996). Based on such observations, it seems plausible that people develop spatial mental models of other types of complex systems, such as the causal patterns under consideration here.…”