SALSA: A Novel Dataset for Multimodal Group Behavior Analysis

Abstract

Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party ) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head/body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA, a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone, accelerometer, bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals' personality as well as their position, head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at http://tev.fbk.eu/salsa.

PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship

“…[5] studied lexical cues from informal texts for recognizing personalities. [6] showed a high correlation of personality with non-verbal behavioral measures such as the amount of speech and physical activity. [7] investigated the physiological correlation of emotion and personality using commercial sensors and found that the emotion-to-personality relationship is better captured by non-linear rather than linear statistics.…”

Detecting Conversational Group Associates from Static Images

“…In the experiment, we used the SALSA dataset Cocktail Party Sequence, which was created by Alameda et al [4]. This dataset records the social interactions among 18 participants in an indoor environment for 500 frames, and is annotated with information about each individual's location, body orientation and group membership.…”

Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in a Triadic Interaction

“…Availability of a large-scale dataset is essential to computationally investigate the nonverbal communication in a data-driven manner. Despite existing datasets that provide measurements for human motion and behaviors [10,38,74,13,2,30], there is no dataset that satisfies the following core requirements for understanding nonverbal human behaviors: (1) capturing 3D full body motion with a broad spectrum of nonverbal cues (including face, body, and hands); (2) capturing signals of naturally interacting groups (more than two people to include attention switching); and (3) collecting the data at scale. The limited availability of datasets motivates us to build a new dataset that contains social interactions among hundreds of interacting groups with a broad spectrum of 3D body motion measurements.…”

“…Social Signal Dataset: How to measure and collect nonverbal signal data is important to pursue a data-driven approach for our goal. However, only few datasets contain socially interacting group motion [2,42,37,58]. The scenes in these datasets are often in a table setup, limiting free body movement and capturing the upper-body only.…”