Aligning speech and co-speech gesture in a constraint-based grammar

Alahverdzhieva, Katya; Lascarides, Ālex; Flickinger, Dan

doi:10.15398/jlm.v5i3.167

Cited by 12 publications

(14 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This research and work by Johnston (1998) inspired grammar-bound models of speech-gesture integration, such as HPSG approaches (see, e.g., Alahverdzhieva & Lascarides 2010, Lücking 2013, to employ nuclear stress for modeling speechgesture integration. These accounts combine a temporal constraint with a phonological constraint differently.…”

Section: Multi-modal Meaningmentioning

confidence: 99%

Multi-modal meaning: An empirically-founded process algebra approach

Rieser

Lawler

2020

S&P

View full text Add to dashboard Cite

Humans communicate with different modalities. We offer an account of multi-modal meaning coordination, taking speech-gesture meaning coordination as a prototypical case. We argue that temporal synchrony (plus prosody) does not determine how to coordinate speech meaning and gesture meaning. Challenging cases are asynchrony and broadcasting cases, which are illustrated with empirical data. We propose that a process algebra account satisfies the desiderata. It models gesture and speech as independent but concurrent processes that can communicate flexibly with each other and exchange the same information more than once. The account utilizes the ψ-calculus, allowing for agents, input-output-channels, concurrent processes, and data transport of typed λ -terms. A multi-modal meaning is produced integrating speech meaning and gesture meaning into one semantic package. Two cases of meaning coordination are handled in some detail: the asynchrony between gesture and speech, and the broadcasting of gesture meaning across several dialogue contributions. This account can be generalized to other cases of multi-modal meaning.

show abstract

Section: Multi-modal Meaningmentioning

confidence: 99%

Multi-modal meaning: An empirically-founded process algebra approach

Rieser

Lawler

2020

S&P

View full text Add to dashboard Cite

show abstract

“…A formal treatment of gestural negation and its grammatical role–in particular, its scope–has been provided by, e.g., Harrison (2010). More generally, recent years have witnessed the development of so-called multimodal grammars , which provide an integrated account of both the spoken and the gestural aspect of human utterance (Johnston et al, 1997; Lascarides and Stone, 2009; Poesio and Rieser, 2009; Alahverdzhieva and Lascarides, 2010; Fricke, 2013). …”

Section: Interactional Aspects Of Communication Already Accepted Amentioning

confidence: 99%

Grammar Is a System That Characterizes Talk in Interaction

Ginzburg

Poesio

2016

Front. Psychol.

View full text Add to dashboard Cite

Much of contemporary mainstream formal grammar theory is unable to provide analyses for language as it occurs in actual spoken interaction. Its analyses are developed for a cleaned up version of language which omits the disfluencies, non-sentential utterances, gestures, and many other phenomena that are ubiquitous in spoken language. Using evidence from linguistics, conversation analysis, multimodal communication, psychology, language acquisition, and neuroscience, we show these aspects of language use are rule governed in much the same way as phenomena captured by conventional grammars. Furthermore, we argue that over the past few years some of the tools required to provide a precise characterizations of such phenomena have begun to emerge in theoretical and computational linguistics; hence, there is no reason for treating them as “second class citizens” other than pre-theoretical assumptions about what should fall under the purview of grammar. Finally, we suggest that grammar formalisms covering such phenomena would provide a better foundation not just for linguistic analysis of face-to-face interaction, but also for sister disciplines, such as research on spoken dialogue systems and/or psychological work on language acquisition.

show abstract

“…In particular, we focus on the handshapes that they use to represent the objects that they are locating in space, what Perniss et al ( 2015 ) term “entity representation.” Focusing on static, rather than moving, objects is expected to facilitate greater precision in our comparison of the handshapes of sign-naïve adults and of signers. The depiction of moving objects runs the risk of gesturers choosing to illustrate the path of the movement and not necessarily the object itself (see similar arguments for gestural ambiguity in Alahverdzhieva and Lascarides, 2010 ). Our focus on static objects avoids this potential confound.…”

Section: Introductionmentioning

confidence: 99%

Using the Hands to Represent Objects in Space: Gesture as a Substrate for Signed Language Acquisition

Janke

Marshall²

2017

Front. Psychol.

View full text Add to dashboard Cite

An ongoing issue of interest in second language research concerns what transfers from a speaker's first language to their second. For learners of a sign language, gesture is a potential substrate for transfer. Our study provides a novel test of gestural production by eliciting silent gesture from novices in a controlled environment. We focus on spatial relationships, which in sign languages are represented in a very iconic way using the hands, and which one might therefore predict to be easy for adult learners to acquire. However, a previous study by Marshall and Morgan (2015) revealed that this was only partly the case: in a task that required them to express the relative locations of objects, hearing adult learners of British Sign Language (BSL) could represent objects' locations and orientations correctly, but had difficulty selecting the correct handshapes to represent the objects themselves. If hearing adults are indeed drawing upon their gestural resources when learning sign languages, then their difficulties may have stemmed from their having in manual gesture only a limited repertoire of handshapes to draw upon, or, alternatively, from having too broad a repertoire. If the first hypothesis is correct, the challenge for learners is to extend their handshape repertoire, but if the second is correct, the challenge is instead to narrow down to the handshapes appropriate for that particular sign language. 30 sign-naïve hearing adults were tested on Marshall and Morgan's task. All used some handshapes that were different from those used by native BSL signers and learners, and the set of handshapes used by the group as a whole was larger than that employed by native signers and learners. Our findings suggest that a key challenge when learning to express locative relations might be reducing from a very large set of gestural resources, rather than supplementing a restricted one, in order to converge on the conventionalized classifier system that forms part of the grammar of the language being learned.

show abstract

Aligning speech and co-speech gesture in a constraint-based grammar

Cited by 12 publications

References 21 publications

Multi-modal meaning: An empirically-founded process algebra approach

Multi-modal meaning: An empirically-founded process algebra approach

Grammar Is a System That Characterizes Talk in Interaction

Using the Hands to Represent Objects in Space: Gesture as a Substrate for Signed Language Acquisition

Contact Info

Product

Resources

About