Three hierarchical positions of deictic gesture in relation to spoken language: a multimodal interaction analysis

Abstract

Taking the action, rather than the utterance or the text, as the unit of analysis, this article isolates different modes, investigating the interdependent relationships, illustrating that the visual mode of gestures can take up a hierarchically equal or a super-ordinate position in addition to the commonly understood sub-ordinate position in relation to the mode of spoken language. Building on McNeill, Birdwhistell, Eco, and Ekman and Friesen, and using a multimodal interaction analytical approach (Norris), I analyse in detail three separate everyday (inter)actions in which a deictic gesture is being performed and spoken language is used by the social actor performing the gesture. With these examples, I build on previous work in multimodal analysis of texts and multimodal interaction analysis, illustrating that the verbal is not necessarily more important than the visual (Kress and Van Leeuwen; Norris; Scollon), demonstrating that verbal and visual modes can be utilized together to (co)produce one message (Van Leeuwen), and showing that a mode utilized by a social actor producing a higher-level discourse structure hierarchically supersedes other modes in interaction (Norris).

The Impact of Literacy-Based Schooling on Learning a Creative Practice: Modal Configurations, Practices and Discourses

“…This second postulate builds on McNeill (1992), where he theorized that "…gestures are an integral part of language as much as are words, phrases and sentences -gesture and language are one system" (Mc Neill, 1992: 2). Just as McNeill asserted that gestures and language form one system, I theorize that all modes in use build one system as explicated in much of my writing but most notably in Norris (2004Norris ( , 2011aNorris ( , 2011bNorris ( , 2013.…”

Functions of three open-palm hand gestures

“…In verbal communication, as described by Norris (2004Norris ( , 2011, linguistic messages acquire meaning in the larger context of interaction. Each interaction is itself included into some higherlevel action and so forth.…”

“…A beat produced with an "unmarked" hand shape (here the open palm) is more prosodic in nature and should be considered as a prosodic unit. Adopting the point of view of modal configuration proposed by Norris (2011), Figure 4 shows the multi-functionality of hand beats which pertain to the mode of gesture as well as to that of prosody which means that rhythm marking is present in the vocal as well as in the visual modality and that beats co-occur preferentially with emphatic stresses rather than other types of stresses in prosody. …”

Reading Multimodal Texts for Learning – a Model for Cultivating Multimodal Literacy

“…These models, and much of earlier analyses of disciplinary discourse as such, are more or less biased towards verbals (writing and speech) rather than seeing all meaning making as multimodal (see Kress, 2009, or Norris, 2011. However, multimodality is ubiquitous in any discipline as well as in learning resources at all levels, and a number of scholars have pointed out the challenges for students regarding the ways that different meaning making resources are used and combined (Danielsson, 2013;Danielsson, accepted;Kress et al, 2001;Kress et al, 2004;Lemke, 1998;Tang & Moje, 2010;Tang et al, 2014, Unsworth;2001).…”