Production and Comprehension of Pantomimes Used to Depict Objects

Abstract

Pantomime, gesture in absence of speech, has no conventional meaning. Nevertheless, individuals seem to be able to produce pantomimes and derive meaning from pantomimes. A number of studies has addressed the use of co-speech gesture, but little is known on pantomime. Therefore, the question of how people construct and understand pantomimes arises in gesture research. To determine how people use pantomimes, we asked participants to depict a set of objects using pantomimes only. We annotated what representation techniques people produced. Furthermore, using judgment tasks, we assessed the pantomimes' comprehensibility. Analyses showed that similar techniques were used to depict objects across individuals. Objects with a default depiction method were better comprehended than objects for which there was no such default. More specifically, tools and objects depicted using a handling technique were better understood. The open-answer experiment showed low interpretation accuracy. Conversely, the forced-choice experiment showed ceiling effects. These results suggest that across individuals, similar strategies are deployed to produce pantomime, with the handling technique as the apparent preference. This might indicate that the production of pantomimes is based on mental representations which are intrinsically similar. Furthermore, pantomime conveys semantically rich, but ambiguous, information, and its interpretation is much dependent on context. This pantomime database is available online: https://dataverse.nl/dataset.xhtml?persistentId=hdl:10411/QZHO6M. This can be used as a baseline with which we can compare clinical groups.

Iconic gestures serve as manual cognates in hearing second language learners of a sign language: An ERP study.

“…At first exposure, they recruit a powerful gestural system that may or may not match the form and meaning of newly encountered signs. These results are in line with more general findings showing that new knowledge is evaluated first in the context of already existing schemas (van Kesteren, Rijpkema, Ruiter, Morris, & Fernández, 2014). These existing schemas are updated after learning, when the acquired signs develop more robust lexical representations and participants distance themselves from their gestures.…”

“…In line with these findings, we hypothesized that if signs with high overlap with gesture are processed more easily compared to signs with low overlap with gesture, reduced amplitude of the N400 component should be observed for the high-overlap condition compared to the low-overlap condition. Second, earlier work has linked N400 amplitude to semantic integration (e.g., van Berkum, Hagoort, & Brown, 1999). It might be easier to integrate an observed high overlap sign (vs. a low overlap sign) with the corresponding preceding word in our paradigm, because of the availability of a gestural schema for the signs with high overlap with gesture.…”

Systematic mappings between semantic categories and types of iconic representations in the manual modality: A normed database of silent gesture

“…Within the manual modality, evidence has shown that people reliably fall back on specific modes of representation when they are asked to express objects in silent gesture. A study showed that after researchers asked a group of adults to represent only with their hands 60 pictures of objects from the Boston Naming Task (Roomer, Hoogerwerf, & Linn, 2011), most concepts were expressed using a default mode of representation that most of the time involved the acting strategy (van Nispen et al, 2014;van Nispen et al, 2017). Interestingly, the concepts depicted through default strategies were also guessed better by a different group of participants (van Nispen et al, 2017).…”

“…The roundness of a ball, the way to operate a saw, the shape of a pyramid-these are all physical sensorimotor attributes that can be grounded in the body for communicative purposes. Individuals build analogical relationships between a real object and a manual form by mapping specific features of their conceptual representations onto an iconic gestural structure (Calbris, 2011;Cooperrider & Goldin-Meadow, 2017;Taub, 2001;van Nispen, van de Sandt-Koenderman, & Krahmer, 2017).…”

“…In this study, we capitalized on the well-established systematicity of silent gestures (Christensen, Fusaroli, & Tylén, 2016;E. Gibson et al, 2013;Goldin-Meadow, So, Ozyürek, & Mylander, 2008;Hall, Mayberry, & Ferreira, 2013;van Nispen, van de Sandt-Koenderman, Mol, & Krahmer, 2014;van Nispen et al, 2017) to investigate whether systematic patterns can also be observed in the silent gestures used to depict individual concepts. As such, we have contributed with a comprehensive normed database of silent gestures produced by a group of participants, providing a detailed description of their forms, their preferred modes of representation, and their degrees of form-meaning transparency (i.e., iconicity ratings) as perceived by a different group of participants.…”