A Roadmap for Technological Innovation in Multimodal Communication Research

Gregori, Alina; Amici, Federica; Brilmayer, Ingmar; Ćwiek, Aleksandra; Fritzsche, Lennart; Fuchs, Susanne; Henlein, Alexander; Herbort, Oliver; Kügler, Frank; Lemanski, Jens; Liebal, Katja; Lücking, Andy; Mehler, Alexander; Nguyen, Kim Tien; Pouw, Wim; Prieto, Pilar; Rohrer, Patrick Louis; Sánchez-Ramón, Paula G.; Schulte-Rüther, Martin; Schumacher, Petra B.; Schweinberger, Stefan R.; Struckmeier, Volker; Trettenbrein, Patrick C.; von Eiff, Celina I.

doi:10.1007/978-3-031-35748-0_30

Cited by 3 publications

(1 citation statement)

References 188 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Automatic recognition and classification tools have the potential to support and speed up labor-intensive manual annotation in multimodal communication research. While historically most such tools focused on acoustic signal processing, recent developments in computer vision research have opened up new possibilities in the visual domain [ 80 ]. Central to these developments is the emergence of reliable pose estimation, the automatic determination of body part locations in images.…”

Section: Methodsmentioning

confidence: 99%

Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study

Bauer,

Kuder,

Schulder

et al. 2024

PLoS ONE

View full text Add to dashboard Cite

This study investigates head nods in natural dyadic German Sign Language (DGS) interaction, with the aim of finding whether head nods serving different functions vary in their phonetic characteristics. Earlier research on spoken and sign language interaction has revealed that head nods vary in the form of the movement. However, most claims about the phonetic properties of head nods have been based on manual annotation without reference to naturalistic text types and the head nods produced by the addressee have been largely ignored. There is a lack of detailed information about the phonetic properties of the addressee’s head nods and their interaction with manual cues in DGS as well as in other sign languages, and the existence of a form-function relationship of head nods remains uncertain. We hypothesize that head nods functioning in the context of affirmation differ from those signaling feedback in their form and the co-occurrence with manual items. To test the hypothesis, we apply OpenPose, a computer vision toolkit, to extract head nod measurements from video recordings and examine head nods in terms of their duration, amplitude and velocity. We describe the basic phonetic properties of head nods in DGS and their interaction with manual items in naturalistic corpus data. Our results show that phonetic properties of affirmative nods differ from those of feedback nods. Feedback nods appear to be on average slower in production and smaller in amplitude than affirmation nods, and they are commonly produced without a co-occurring manual element. We attribute the variations in phonetic properties to the distinct roles these cues fulfill in turn-taking system. This research underlines the importance of non-manual cues in shaping the turn-taking system of sign languages, establishing the links between such research fields as sign language linguistics, conversational analysis, quantitative linguistics and computer vision.

show abstract

Section: Methodsmentioning

confidence: 99%

Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study

Bauer,

Kuder,

Schulder

et al. 2024

PLoS ONE

View full text Add to dashboard Cite

show abstract

An Outlook for AI Innovation in Multimodal Communication Research

Henlein,

Bauer,

Bhattacharjee

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Non‐verbal effecting – animal research sheds light on human emotion communication

Schirmer,

Croy,

Liebal

et al. 2024

Biological Reviews

View full text Add to dashboard Cite

Cracking the non‐verbal “code” of human emotions has been a chief interest of generations of scientists. Yet, despite much effort, a dictionary that clearly maps non‐verbal behaviours onto meaning remains elusive. We suggest this is due to an over‐reliance on language‐related concepts and an under‐appreciation of the evolutionary context in which a given non‐verbal behaviour emerged. Indeed, work in other species emphasizes non‐verbal effects (e.g. affiliation) rather than meaning (e.g. happiness) and differentiates between signals, for which communication benefits both sender and receiver, and cues, for which communication does not benefit senders. Against this backdrop, we develop a “non‐verbal effecting” perspective for human research. This perspective extends the typical focus on facial expressions to a broadcasting of multisensory signals and cues that emerge from both social and non‐social emotions. Moreover, it emphasizes the consequences or effects that signals and cues have for individuals and their social interactions. We believe that re‐directing our attention from verbal emotion labels to non‐verbal effects is a necessary step to comprehend scientifically how humans share what they feel.

show abstract

A Roadmap for Technological Innovation in Multimodal Communication Research

Cited by 3 publications

References 188 publications

Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study

Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study

An Outlook for AI Innovation in Multimodal Communication Research

Non‐verbal effecting – animal research sheds light on human emotion communication

Contact Info

Product

Resources

About