Analyzing Multifunctionality of Head Movements in Face-to-Face Conversations Using Deep Convolutional Neural Networks

Abstract

A functional head-movement corpus and convolutional neural networks (CNNs) for detecting head-movement functions are presented for analyzing the multiple communicative functions of head movements in multiparty face-to-face conversations. First, focusing on the multifunctionality of head movements, i.e., that a single head movement can simultaneously perform multiple functions, this paper defines 32 nonmutually-exclusive function categories, whose genres are speech production, eliciting and giving feedback, turn management, and cognitive and affect display. To represent and capture arbitrary multifunctional structures, our corpus employs multiple binary codes and logical-sum-based aggregations of multiple coders' judgments. A corpus analysis targeting four-party Japanese conversations revealed multifunctional patterns in which the speaker modulates multiple functions, such as emphasis and eliciting listeners' responses, through rhythmic head movements, and listeners express various attitudes and responses through continuous back-channel head movements. This paper proposes CNN-based binary classifiers for detecting each of the functions from the angular velocity of the head pose and the presence or absence of utterances. The experimental results showed that the recognition performance varies greatly, from approximately 30% to 90% in terms of the F-score, depending on the function category, and the performance was positively correlated with the amount of data and inter-coder agreement. In addition, we noted a tendency toward overdetection that added more functions to those originally in the corpus. The analyses and experiments confirm that our approach is promising for studying the multifunctionality of head movements. INDEX TERMS deep neural networks, gesture recognition, meeting analysis, multimodal sensor, nonverbal behaviors, social signal processing This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.

Observational and Accelerometer Analysis of Head Movement Patterns in Psychotherapeutic Dialogue

“…There are multiple functionalities for head movements. For example, Otsuka defined 32 function categories including 13 related to speaking, 12 related to reacting, and seven related to other functions [ 40 ]. They argued that there is no fixed one-to-one relation between kinetic features and functions.…”

Prediction of Interlocutors’ Subjective Impressions Based on Functional Head-Movement Features in Group Meetings

“…Otsuka & Tsumori proposed a method to estimate the communicative functions of head movements from the time series of head pose angles, focusing on the communicative functions of head movements rather than the types of kinetic characteristics of head movements [20]. The head movement functions defined by the above authors include emphasis and rhythm related to speech, back channeling and thinking related to reaction, among others.…”

“…Among the nonverbal behaviors that occur during group meetings, head movements are known to serve a variety of functions [20]. For example, speakers use their head movements to emphasize their speech or to confirm listeners' responses, and listeners use their head movements as a sign of agreement with or response to speakers.…”

“…Thus, head movements have multiple functions, and a single head movement may have multiple meanings. Focusing on the diversity and ambiguity of the head-movement functions, Otsuka & Tsumori defined the functional categories of head movements and created the Functional Head-Movement Corpus (hereafter referred to as the FHM Corpus), which nonexclusively annotates the multiple functions performed through head movements on each video frame in group meetings [20]. Using this corpus, they developed convolutional neural networks (CNNs) to detect the presence/absence of each of the ten most frequent head-movement functions from the time series of head poses and the presence/absence of utterances.…”

“…In the proposed prediction model, interlocutors' head movements and the ten most frequent head-movement functions (in Table 1) for each video frame are first detected by CNNs [20] from the time series of the head pose angle and the presence/absence of utterance. Then, from these detection results, we newly define and calculate the functional head-movement features, including the occurrence rate and composition ratio of the head-movement functions at every two-minute interval in the meeting.…”