NovA: Automated Analysis of Nonverbal Signals in Social Interactions

Abstract

Previous studies have shown that the success of interpersonal interaction depends not only on the contents we communicate explicitly, but also on the social signals that are conveyed implicitly. In this paper, we present NovA (NOnVerbal behavior Analyzer), a system that analyzes and facilitates the interpretation of social signals conveyed by gestures, facial expressions and others automatically as a basis for computer-enhanced social coaching. NovA records data of human interactions, automatically detects relevant behavioral cues as a measurement for the quality of an interaction and creates descriptive statistics for the recorded data. This enables us to give a user online generated feedback on strengths and weaknesses concerning his social behavior, as well as elaborate tools for offline analysis and annotation.

The hybrid Agent MARCO

“…We believe that our hybrid agent enables us to study how virtual and robotic parts can be best combined to achieve and study emotional contagion between man and machine in the context of chess. Therefore, we aim to analyze for differences in a human player's nonverbal reactions to the hybrid agent using the "NovA" framework [2].…”

“…The hardware used comprises a 5.6 inch pan-tilt-roll display to present the virtual agent's face, a robotic arm to move the chess pieces, and a digital chess board (DGT USB Rosewood) with a digital chess clock. A Kinect RGBD camera is used by the "NovA -Nonverbal Behavior Analyzer" [2] to track the human player's nonverbal behavior for online and offline analysis. The pan-tilt-roll display features a physical resolution of 640×480 pixels at 16bit color depth.…”

Augmenting Social Interactions

“…However, the human learner is usually not given any explicit feedback on her or his performance before the end of the interaction. For instance, the TARDIS system [3] enables the learners to inspect previous interactions conducted with a virtual character serving as a recruiter in a job interview. In this paper, we present an approach that provides realtime feedback on social behaviours: directives are automatically generated by a computer system while the user is engaging in a social interaction.…”

The hybrid agent MARCO

“…For tracking and analysis of the human players' nonverbal behaviors during gameplay the "NovA -Nonverbal Behavior Analyzer" is integrated into our setup [3]. In combination with the Microsoft Kinect sensor this framework allows for the automatic detection, recording, and offline analysis of head orientation and body posture.…”