Challenges of multimodality: Language and the body in social interaction

Abstract

This article reflects on recent challenges emerging from the study of language and the body in social interaction. There is a general interest in language and the body across disciplines that has invited a reconceptualization of the broader issues relative to action, cognition, culture, knowledge, social relations and identities, spatiality and temporality. The study of social interaction focuses on how multimodal resources – including language and bodily movements – are holistically and situatedly used in building human action. This article discusses some consequences and challenges of putting the body at the center of attention: it repositions language as one among other modalities, and invites us to consider the involvement of entire bodies in social interaction, overcoming a logo‐centric vision of communication, as well as a visuo‐centric vision of embodiment. These issues are developed through a series of conversation analytic studies, firstly of classic topics in linguistics like deixis, then of more recent topics, such as mobility and sensoriality.

Processing language in face-to-face conversation: Questions with gestures get faster responses

“…The primary site of human language use is face-to-face conversation, suggesting that human language should be conceptualized as a fundamentally multimodal phenomenon (e.g., Bavelas & Chovil, 2000;Clark, 1996;Goldin-Meadow, 2003;Kendon, 2004Kendon, , 2014Levinson & Holler, 2014;Mondada, 2016;McNeill, 1992). Bodily signals, in particular, manual gestures, add a significant amount of meaning to what is being said.…”

An Appeal for a Methodological Fusion of Conversation Analysis and Experimental Psychology

“…CA's generalizations have tended to focus on the systematic conventionalization and grammaticalization of recurrent patterns of talk. However, video-based studies of embodied interaction tend instead to emphasize the evidential importance of specific, situated occasions of interaction (Mondada, 2016). The "embodied turn" in interaction research (Nevile, 2015) toward exploring visible, copresent social action is productively challenging many of CA's foundational generalizations, which were mostly derived from analyses of talk on the phone (Couper- Kuhlen, 2010).…”

Social Moments: A Perspective on Interaction for Social Robotics

“…Altogether, a large set of events across multiple modalities can intertwine to create social meanings, given a specific context (Mondada, 2016). A good example is the McGurk effect (McGurk and MacDonald, 1976), where the visual and auditory channels are integrated during language perception.…”

“…Using neuromorphic processing of visual inputs, studies have achieved categorization of objects in less than 160 ms (Wang et al, 2017) or triggered robot responses in 4 cycles of a periodic event (Wiles et al, 2010). In addition, the interpretation of social moments requires the integration of information across multiple modalities, as multiple modalities contribute to the generation of social meaning (Mondada, 2016). Finally, as the social meanings interpreted from a social moment can affect long-term social interactions, robots need to be able to integrate social information obtained at short timescales into their cognitive architecture [see Lemaignan et al (2017)] and memory to be able to represent the context against which future responses will occur.…”