Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in a Triadic Interaction

Abstract

We present a new research task and a dataset to understand human social interactions via computational methods, to ultimately endow machines with the ability to encode and decode a broad channel of social signals humans use. This research direction is essential to make a machine that genuinely communicates with humans, which we call Social Artificial Intelligence. We first formulate the "social signal prediction" problem as a way to model the dynamics of social signals exchanged among interacting individuals in a data-driven way. We then present a new 3D motion capture dataset to explore this problem, where the broad spectrum of social signals (3D body, face, and hand motions) are captured in a triadic social interaction scenario. Baseline approaches to predict speaking status, social formation, and body gestures of interacting individuals are presented in the defined social prediction framework.

Statistics‐based Motion Synthesis for Social Conversations

“…Although their models are trained for a single subject, their frameworks are good starting points for designing dyadic conversation models. In additional to dyadic conversation, three-party conversations are also studied in recent work [JDZD19;dCYS*19;JSCS19]. While motion graph approaches are good at maintaining styles and high motion quality in terms of dynamics and smoothness from the captured dataset, learning based methods have the potential for fast inference after models are trained, and may find more diverse results.…”

EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset

“…Yonetani et al (2016) collect a human interaction dataset and study action and reaction recognition. Joo et al (2019) present a task and a 3D motion dataset to understand human social interactions. introduce a dual relation modeling framework for egocentric human interactions using vision signals.…”

Perceiving Humans: From Monocular 3D Localization to Social Distancing

“…We aim to capture social interactions among people and monitor social distancing from visual cues. Related works include the broad field of behavioral science [94]. Here we focus on the subfield called proxemics, which investigates how people use and organize the space they share with others [25], [49].…”

“…Recently, deep learning approaches have been adopted to understand social interactions under a different perspective. Joo et al [94] learn to predict behavioral cues of a target person (e.g., body orientation) from the position and orientation of another person. They learn the dynamics between social interactions in a data-driven manner, laying the foundations for deep learning to be applied in the field of behavioral science.…”