Nonverbal Social Sensing: What Social Sensing Can and Cannot Do for the Study of Nonverbal Behavior From Video

Abstract

The study of nonverbal behavior (NVB), and in particular kinesics (i.e., face and body motions), is typically seen as cost-intensive. However, the development of new technologies (e.g., ubiquitous sensing, computer vision, and algorithms) and approaches to study social behavior [i.e., social signal processing (SSP)] makes it possible to train algorithms to automatically code NVB, from action/motion units to inferences. Nonverbal social sensing refers to the use of these technologies and approaches for the study of kinesics based on video recordings. Nonverbal social sensing appears as an inspiring and encouraging approach to study NVB at reduced costs, making it a more attractive research field. However, does this promise hold? After presenting what nonverbal social sensing is and can do, we discussed the key challenges that researchers face when using nonverbal social sensing on video data. Although nonverbal social sensing is a promising tool, researchers need to be aware of the fact that algorithms might be as biased as humans when extracting NVB or that the automated NVB coding might remain context-dependent. We provided study examples to discuss these challenges and point to potential solutions.

From low invasiveness to high control: how artificial intelligence allows to generate a large pool of standardized corpora at a lesser cost

“…Third, the presented innovative approach contributes to corpora development by facilitating data sharing. Although not specifically addressed in this paper, the traditional approach also has issues related to data protection, hence adding another challenge to the progress of this research (see Renier et al, 2021). Laws related to data privacy require researchers to collect consent forms from participants to use their videos.…”

“The interviewer is a machine!” Investigating the effects of conventional and technology‐mediated interview methods on interviewee reactions and behavior

“…Organizations might also decide to focus on the verbal content of the interviews by providing only the audio‐recordings, or even the transcripts (see Lukacik et al, 2022). To go further in the automatization of the pre‐screening step, interviewee evaluation can also be delegated to algorithms designed to automatically extract data from videotapes (Hickman et al, 2022; Langer et al, 2019; Lukacik et al, 2022; Rao et al, 2017; Rasipuram & Jayagopi, 2018; Renier et al, 2021).…”

Nonverbal Communication as a Communication Strategy in Hotel Job Interview: A Narrative Inquiry Study

“…The way how the employee reacts or responses towards client's request will affect the output of clients' attitude (Shadaydeh et al, 2021). The employee needs to have proper manner in dealing with the customer needs by employing appropriate nonverbal techniques, such as open body language, good eye contact, and smiling (Novotny et al, 2021) to show friendliness (Renier et al, 2021) and kindness (Zand et al, 2020). With proper nonverbal technique, the client or customer will start to enjoy the conversation and indirectly it will impact to listener's attitude and emotions (Anuradha & Singh, 2020).…”