Graphical models for social behavior modeling in face-to face interaction

Abstract

International audienceThe goal of this paper is to model the coverbal behavior of a subject involved in face-to-face social interactions. For this end, we present a multimodal behavioral model based on a Dynamic Bayesian Network (DBN). The model was inferred from multimodal data of interacting dyads in a specific scenario designed to foster mutual attention and multimodal deixis of objects and places in a collaborative task. The challenge for this behavioral model is to generate coverbal actions (gaze, hand gestures) for the subject given his verbal productions, the current phase of the interaction and the perceived actions of the partner. In our work, the structure of the DBN was learned from data, which revealed an interesting causality graph describing precisely how verbal and coverbal human behaviors are coordinated during the studied interactions. Using this structure, DBN exhibits better performances compared to classical baseline models such as Hidden Markov Models (HMMs) and Hidden Semi-Markov Models (HSMMs). We outperform the baseline in both measures of performance, i.e. interaction unit recognition and behavior generation. DBN also reproduces more faithfully the coordination patterns between modalities observed in ground truth compared to the baseline models

Learning off-line vs. on-line models of interactive multimodal behaviors with recurrent neural networks

“…The LSTM and Bi-LSTM can automatically learn contextual variables from the interaction scenario. In order to compare the efficiency of the methods, actions (FX and GT) generated offline by BiLSTM are first compared with HMM [5] and DBN [6]. In addition, online predictions of the actions by LSTM are also compared with short-term Viterbi decoding of HMM and online filter prediction of DBN.…”

“…The dataset 1 used as interactive data in this paper has been collected by Mihoub et al [6]. This face-to-face interaction involves an instructor and a manipulator who performed a collaborative task called "put that there".…”

“…Mihoub et al [5] estimated interaction units using Hidden Markov Models (HMM) to generate the gaze of an interlocutor from his own speech activity together with the gaze and speech activity of his partner. Mihoub et al [6] further showed that DBN outperform both full-and semi-HMM in predicting co-verbal behaviors in a "put that there" game. Actually, few works have been devoted to the modeling of joint behaviors while incredible amount of research have been successfully dealing with recognition of human activities from multimodal behaviors [7] [8] and vice-versa [9] [10].…”

Social Intelligence Modeling using Wearable Devices

“…conditional probability distributions do not change over time), a DBN can be sufficiently depicted by a 2-timeslice structure called 2-timeslice Bayesian Network (2-TBN) (see example in Figure 7) [14]. Due to their multiple characteristics, DBNs have been used successfully in many domains; in particular, they represent an attractive and a powerful formalism for modeling dynamics of human behavior in social interactions [23,36].…”

“…For information, Weka Java API [18] has been used to realize the clustering task. After many empirical tests (9,18,27,36), we chose the optimal number of 27 clusters, which corresponds, to 27 specific CS.…”