Monitoring of dyadic conversations : a social signal processing approach

Abstract

This work presents a real-time system that analyzes non-verbal audio and visual cues to quantitatively assess sociometries from on-going two-person conversations.The system non-invasively captures audio and video/depth data from lapel microphones and Microsoft Kinect devices respectively to extract non-verbal speech and visual cues. The system leverages these non-verbal cues to quantitatively assess speaking mannerisms of each participant. The speech and visual cues are incorporated as features in machine learning algorithms to quantify various aspects of social behavior including Interest, Dominance, Politeness, Friendliness, Frustration, Empathy, Respect, . Confusion, Hostility and Agreement. The most relevant speech and visual cues are selected by forward feature selection. The system is trained and tested on two carefully annotated corpora, i.e., an Audio Corpus (AC) and Audio-Visual Corpus (AVC) comprising brief two-person dialogs (in English). Numerical tests through leave-one-person-out cross-validation indicate that the accuracy of the algorithms for inferring the sociometries is in the range of 50% -86% for AC and 62% -92% for AVC. To test the robustness of the proposed approach, the audio • data from both corpora are combined, and a classifier is trained on this mixed data set. Despite the significant differences in the recording conditions of the AC and NANYANG TECHNOLOGICAL UNIVERSITY SINGAPORE