Agreement and disagreement classification of dyadic interactions using vocal and gestural cues

Khaki, Hossein; Bozkurt, Elif; Erzin, Engin

doi:10.1109/icassp.2016.7472180

Cited by 4 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is an increasing body of literature on detecting different social attitudes such as dominance, agreement, or engagement in interactions (Dermouche and Pelachaud 2019a;Khaki et al 2016;Bee et al 2009). These tackle the interaction from video recordings and could be applicable to VCs on 2D displays.…”

Section: Detecting Social Attitudesmentioning

confidence: 99%

Immersive machine learning for social attitude detection in virtual reality narrative games

2022

View full text Add to dashboard Cite

People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program Non-Player Characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We collaborated with two game studios to develop an immersive machine learning (ML) pipeline for detecting social engagement. We collected data from participants-NPC interaction in VR, which was then annotated in the same immersive environment. Game design is a creative process and it is vital to respect designer’s creative vision and judgement. We therefore view annotation as a key part of the creative process. We trained a reinforcement learning algorithm (PPO) with imitation learning rewards using raw data (e.g. head position) and socially meaningful derived data (e.g. proxemics); we compared different ML configurations including pre-training and a temporal memory (LSTM). The pre-training and LSTM configuration using derived data performed the best (84% F1-score, 83% accuracy). The models using raw data did not generalise. Overall, this work introduces an immersive ML pipeline for detecting social engagement and demonstrates how creatives could use ML and VR to expand their ability to design more engaging experiences. Given the pipeline’s results for social engagement detection, we generalise it for detecting human-defined social attitudes.

show abstract

Section: Detecting Social Attitudesmentioning

confidence: 99%

Immersive machine learning for social attitude detection in virtual reality narrative games

2022

View full text Add to dashboard Cite

show abstract

“…In our previous study [23], we investigate a multimodal twoclass dyadic interaction type (DIT) estimation approach of agreement and disagreement classes from speech and motion DIT classifier Figure 2: Dyadic Interaction Type -Continuous Emotion Recognition (DIT-CER) system.…”

Section: Dyadic Interaction Type Estimationmentioning

confidence: 99%

Use of Agreement/Disagreement Classification in Dyadic Interactions for Continuous Emotion Recognition

Khaki

Erzin

2016

Interspeech 2016

Self Cite

View full text Add to dashboard Cite

Natural and affective handshakes of two participants define the course of dyadic interaction. Affective states of the participants are expected to be correlated with the nature or type of the dyadic interaction. In this study, we investigate relationship between affective attributes and nature of dyadic interaction. In this investigation we use the JESTKOD database, which consists of speech and full-body motion capture data recordings for dyadic interactions under agreement and disagreement scenarios. The dataset also has affective annotations in activation, valence and dominance (AVD) attributes. We pose the continuous affect recognition problem under agreement and disagreement scenarios of dyadic interactions. We define a statistical mapping using the support vector regression (SVR) from speech and motion modalities to affective attributes with and without the dyadic interaction type (DIT) information. We observe an improvement in estimation of the valence attribute when the DIT is available. Furthermore this improvement sustains even we estimate the DIT from the speech and motion modalities of the dyadic interaction.

show abstract

Expressing Agreement in Swedish and Chinese: A Case Study of Communicative Feedback in First-Time Encounters

Gander

Lindström

Gander

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Agreement and disagreement classification of dyadic interactions using vocal and gestural cues

Cited by 4 publications

References 17 publications

Immersive machine learning for social attitude detection in virtual reality narrative games

Immersive machine learning for social attitude detection in virtual reality narrative games

Use of Agreement/Disagreement Classification in Dyadic Interactions for Continuous Emotion Recognition

Expressing Agreement in Swedish and Chinese: A Case Study of Communicative Feedback in First-Time Encounters

Contact Info

Product

Resources

About