Automatic Analysis of Bodily Social Signals

Abstract

The human body plays an important role in face-to-face interactions (Knapp & Hall, 2010;McNeill, 1992). We use our bodies to regulate turns, to display attitudes and to signal attention (Scheflen, 1964). Unconsciously, the body also reflects our affective and mental states (Ekman & Friesen, 1969). There is a long history of research into the bodily behaviors that correlate with the social and affective state of a person, in particular in interaction with others (Argyle, 2010;Dittmann, 1987;Mehrabian, 1968). We will refer to these behaviors as bodily social signals. These social and affective cues can be detected and interpreted by observing the human body's posture and movement (Harrigan, 2008;Kleinsmith & Bianchi-Berthouze, 2013). Automatic observation and analysis has applications such as the detection of driver fatigue and deception, the analysis of interest and mood in interactions with robot companions, and in the interpretation of higher-level phenomena such as mimicry and turn-taking.In this chapter, we will discuss various bodily social signals, and how to analyze and recognize them automatically. Human motion can be studied on many levels, from the physical level involving muscles and joints, to the level of interpreting a person's fullbody actions and intentions (Poppe, 2007(Poppe, , 2010Jiang et al., 2013). We will focus on automatically analyzing movements with a relatively short time scale, such as a gesture or posture shift. In the first section, we will discuss the different ways of measurement and coding, both from motion capture data and images and video. The recorded data can subsequently be interpreted in terms of social signals. In the second section, we address the automatic recognition of several bodily social signals. We will conclude the chapter with a discussion of challenges and directions of future work. Measurement of Body MotionBody movement can be observed and described quantitatively, for example, in terms of joint rotations or qualitatively with movement labels. While social signals are typically detected and identified as belonging to a certain category, body motion is typically described quantitatively. Therefore, the detection of bodily social signals is often based on a quantitative representation of the movement. From the perspective of computation, body motion is most conveniently recorded and measured using motion capture (mocap) devices. However, their obtrusive nature, cost, and the fact that they typically cannot be used outside the laboratory has limited their employment. Therefore, many researchers available at https://www.cambridge.org/core/terms.

MUMBAI: multi-person, multimodal board game affect and interaction analysis dataset

“…In addition to these features, machine learning researchers use representations that rely on feature extraction (e.g. via deep learning), which are more difficult to interpret than movement and posture [58].…”

Mining Bodily Cues to Deception

“…Vice versa, automated analysis can be more easily used for theory verification and development. Unfortunately, for the analysis of body movement, there is no such commonly agreed on quantitative representation system (Poppe, 2017 ). Bodily expressions are arguably more complex to represent than facial expressions because of the larger number of degrees of freedom which gives a wide range of possible body poses.…”

Nonverbal Social Sensing: What Social Sensing Can and Cannot Do for the Study of Nonverbal Behavior From Video

“…In this paper, we focused on kinesics and the use of nonverbal social sensing based on video recordings (see Poppe, 2017 for an application of nonverbal social sensing beyond video recordings). Kinesics refers to two categories of NVB: (1) gesture and posture and (2) face and eye behavior ( Vinciarelli et al, 2009a ; the latter is also referred to as gaze, Harrigan, 2005 , p. 137).…”