Multi-modal meaning: An empirically-founded process algebra approach

Abstract

Humans communicate with different modalities. We offer an account of multi-modal meaning coordination, taking speech-gesture meaning coordination as a prototypical case. We argue that temporal synchrony (plus prosody) does not determine how to coordinate speech meaning and gesture meaning. Challenging cases are asynchrony and broadcasting cases, which are illustrated with empirical data. We propose that a process algebra account satisfies the desiderata. It models gesture and speech as independent but concurrent processes that can communicate flexibly with each other and exchange the same information more than once. The account utilizes the ψ-calculus, allowing for agents, input-output-channels, concurrent processes, and data transport of typed λ -terms. A multi-modal meaning is produced integrating speech meaning and gesture meaning into one semantic package. Two cases of meaning coordination are handled in some detail: the asynchrony between gesture and speech, and the broadcasting of gesture meaning across several dialogue contributions. This account can be generalized to other cases of multi-modal meaning.

Laughter Meaning Construction and Use in Development: Children and Spoken Dialogue Systems

“…In addition, we observed that laughter does "interrupt" and does overlap with own and others' speech, invalidating thus the claim that laughter occurs exclusively during pauses and at phrase boundaries (Laughter punctuation effect [82]). The patterns observed resemble therefore those reported for manual gesture in relation to speech [86,2].…”

“…-Laughter can co-occur with speech from the laugher herself and from the partner, and can interrupt speech utterances. -Its positioning in relation to the argument can be rather free, occurring most often after, but also during or before the argument it is related to; mirroring patterns observed in manual gestures [86]. -In order to interpret the laughter functions performed, acoustics in itself is not sufficient, since they seem rather characterised by a language-dependent cluster of features: positioning in relation to speech, in relation to others' laughter, position in relation to the laughable, characteristics of the laughable, and contingent gaze patterns from the laugher and from the partner.…”

A Roadmap for Technological Innovation in Multimodal Communication Research

“…In order to capture these features of spoken language, ASR systems have to be developed "more impurely" from a phonetic and incremental point of view, including that communication researchers agree on a useful transcription system (for humans and computers) -and provide big amounts of transcribed data. -While temporal synchrony is an important aspect of multiple signals in faceto-face interactions, it yet does not fully determine coordination of semantic meaning of those signals [162]. While temporal alignment is an observable, measurable feature of multi-channel communication, semantic integration involves interpretation [94].…”