A Coding System with Independent Annotations of Gesture Forms and Functions During Verbal Communication: Development of a Database of Speech and GEsture (DoSaGE)

Abstract

Gestures are commonly used together with spoken language in human communication. One major limitation of gesture investigations in the existing literature lies in the fact that the coding of forms and functions of gestures has not been clearly differentiated. This paper first described a recently developed Database of Speech and GEsture (DoSaGE) based on independent annotation of gesture forms and functions among 119 neurologically unimpaired right-handed native speakers of Cantonese (divided into three age and two education levels), and presented findings of an investigation examining how gesture use was related to age and linguistic performance. Consideration of these two factors, for which normative data are currently very limited or lacking in the literature, is relevant and necessary when one evaluates gesture employment among individuals with and without language impairment. Three speech tasks, including monologue of a personally important event, sequential description, and story-telling, were used for elicitation. The EUDICO Linguistic ANnotator (ELAN) software was used to independently annotate each participant’s linguistic information of the transcript, forms of gestures used, and the function for each gesture. About one-third of the subjects did not use any co-verbal gestures. While the majority of gestures were non-content-carrying, which functioned mainly for reinforcing speech intonation or controlling speech flow, the content-carrying ones were used to enhance speech content. Furthermore, individuals who are younger or linguistically more proficient tended to use fewer gestures, suggesting that normal speakers gesture differently as a function of age and linguistic performance.

A Comparison of Coverbal Gesture Use in Oral Discourse Among Speakers With Fluent and Nonfluent Aphasia

“…Using Kong, Law, Kwan, et al's (2015) coding system, how do the distributions of gesture forms and functions differ across fluent PWA, nonfluent PWA, and unimpaired control speakers? 3.…”

“…The criterion of annotating gestures in the present study follows that in Kong, Law, Kwan, et al (2015). To be specific, a unit of gesture was defined as the period between hand(s) starting to move until returning to the original resting position (McNeill, 1992).…”

“…In cases in which a specific gesture was judged to serve more than one purpose, the final assignment was based on the primary function in relation to the language content (see details of coding criteria in Appendix A). Kong, Law, Kwan, et al's (2015) annotation scheme on gesture functions partly overlapped with the gesture-coding framework for children (Colletta, Kunene, Venouil, Kaufmann, & Simon, 2009) and challenged a previous assumption that one form of gesture could only serve one function (see Scharp, Tompkins, & Iverson, 2007).…”

“…Kendon's (1988) classification of gestures emphasized their forms and types but failed to address the functional roles they play in communication. To provide a more systematic method for gesture annotation, Kong, Law, Kwan, Lai, and Lam (2015) proposed a coding system to independently annotate gesture forms and functions. To be more specific, there are six forms of gestures modified on the basis of the classification by Ekman and Friesen (1969), Mather (2005), and McNeill (1992), including contentcarrying gestures: (a) iconic gestures that specify the spatial and dynamic features of concrete objects or actions, (b) metaphoric gestures that specify abstract ideas, (c) deictic gestures that encompass pointing to a physically present referent or to an absent referent, and (d) emblems that are culturally conformed and universally understood by a community and non-content-carrying gestures: (e) beats that refer to hand-flicking movements following the speech rhythm and (f ) nonidentifiable gestures that include ambiguous gestures or those that cannot be classified into the aforementioned categories.…”

“…To be more specific, there are six forms of gestures modified on the basis of the classification by Ekman and Friesen (1969), Mather (2005), and McNeill (1992), including contentcarrying gestures: (a) iconic gestures that specify the spatial and dynamic features of concrete objects or actions, (b) metaphoric gestures that specify abstract ideas, (c) deictic gestures that encompass pointing to a physically present referent or to an absent referent, and (d) emblems that are culturally conformed and universally understood by a community and non-content-carrying gestures: (e) beats that refer to hand-flicking movements following the speech rhythm and (f ) nonidentifiable gestures that include ambiguous gestures or those that cannot be classified into the aforementioned categories. With reference to the content of the corresponding language output, each gesture is also classified into one of the following eight functions, namely (a) providing additional information to a message conveyed (Goldin-Meadow, 2003); (b) enhancing the speech content (Beattie & Shovelton, 2000); (c) providing alternative means of communication (Le May, David, & Thomas, 1988); (d) guiding and controlling the flow of speech (Jacobs & Garnham, 2007); (e) reinforcing the intonation or prosody of speech (Kong, Law, Kwan, et al, 2015); (f ) assisting lexical retrieval (Krauss & Hadar, 1999); (g) assisting sentence reconstruction (Alibali, Kita, & Young, 2000); and (h) serving other nonspecific or noncommunicative functions, such as assisting the process of verbal dialogues without conveying topical information, regulating and organizing the process of coming together with language content, or the back-and-forth nature of speaking and listening. In cases in which a specific gesture was judged to serve more than one purpose, the final assignment was based on the primary function in relation to the language content (see details of coding criteria in Appendix A).…”