MUMBAI: multi-person, multimodal board game affect and interaction analysis dataset

Abstract

Board games are fertile grounds for the display of social signals, and they provide insights into psychological indicators in multi-person interactions. In this work, we introduce a new dataset collected from four-player board game sessions, recorded via multiple cameras, and containing over 46 hours of visual material. The new MUMBAI dataset is extensively annotated with emotional moments for all game sessions. Additional data comes from personality and game experience questionnaires. Our four-person setup allows the investigation of non-verbal interactions beyond dyadic settings. We present three benchmarks for expression detection and emotion classification and discuss potential research questions for the analysis of social interactions and group dynamics during board games.

Didn't see that coming: a survey on non-verbal social human behavior forecasting

“…Usually, third-view datasets consist of structured interactions where participants need to follow basic directives which favor spontaneous and fluent interactions. Despite the fact that conversations are the most common interaction structure, there are datasets which aim at fostering specific social signals like leadership, competitiveness, empathy, or affect, and therefore engage the participants in competitive/cooperative scenarios (Hung and Chittaranjan, 2010;Sanchez-Cortes et al, 2012;Rehg et al, 2013;Ringeval et al, 2013;Vella and Paggio, 2013;Bambach et al, 2015;Salter et al, 2015;Edwards et al, 2016;Beyan et al, 2016;Georgakis et al, 2017;Doyran et al, 2021;. Other datasets, instead, record in-the-wild interactions during the so-called cocktail parties (Alameda-Pineda et al, 2016;Cabrera-Quiros et al, 2018) and represent very interesting benchmarks to study group dynamics.…”

“…The latter is considerably less frequent due to its tedious manual annotation process (McCowan et al, 2005;Douglas-Cowie et al, 2007;McKeown et al, 2010;Lücking et al, 2012;Vella and Paggio, 2013;Vandeventer et al, 2015;Naim et al, 2015;Chou et al, 2017;Paggio and Navarretta, 2017;Cafaro et al, 2017;Joo et al, 2019b;Kossaifi et al, 2019;Chen et al, 2020;Khan et al, 2020;. The most frequent low-level annotations that the datasets provide are the participants' body poses and facial expressions (Douglas-Cowie et al, 2007;Rehg et al, 2013;Bilakhia et al, 2015;Vandeventer et al, 2015;Naim et al, 2015;Edwards et al, 2016;Cafaro et al, 2017;Feng et al, 2017;Georgakis et al, 2017;Paggio and Navarretta, 2017;Bozkurt et al, 2017;Andriluka et al, 2018;von Marcard et al, 2018;Mehta et al, 2018;Lemaignan et al, 2018;Joo et al, 2019b;Kossaifi et al, 2019;Schiphorst et al, 2020;Doyran et al, 2021;. Given their annotation complexity, they are usually automatically retrieved with tools like OpenPose (Cao et al, 2019), and manually fixed or discarded.…”

“…Indeed, some of the datasets have been complementary annotated and added in posterior studies. As a result, most common high-level labels consist of elicited emotions (McCowan et al, 2005;Douglas-Cowie et al, 2007;van Son et al, 2008;McKeown et al, 2010;Naim et al, 2015;Vandeventer et al, 2015;Chou et al, 2017;Paggio and Navarretta, 2017;Maman et al, 2020;Doyran et al, 2021), action labels (Soomro et al, 2012;Yonetani et al, 2016;Silva et al, 2018;Abebe et al, 2018;Carreira et al, 2019;Zhao et al, 2019;Schiphorst et al, 2020;Monfort et al, 2020;Martín-Martín et al, 2021), and social cues/signals (Hung and Chittaranjan, 2010;Sanchez-Cortes et al, 2012;Ringeval et al, 2013;Vandeventer et al, 2015;Shukla et al, 2016;Bozkurt et al, 2017;Cafaro et al, 2017;Feng et al, 2017;Lemaignan et al, 2018;Cabrera-Quiros et al, 2018;Celiktutan et al, 2019;Chen et al, 2020;Maman et al, 2020).…”

Correlating Facial Expressions and Subjective Player Experiences in Competitive Hearthstone

“…Facial expression analysis and video games have been combined in multiple studies discussing various topics, such as affective gaming [22,27], game personalisation [7], player affect evaluation [24,28] and alternative gameplay mechanisms [25]. Recently, Doyran et al [9] published a rich dataset that enables multi-modal, multi-player affect and interaction analysis through capturing the facial expressions of board game players.…”

Multimodal Music Emotion Recognition Method Based on the Combination of Knowledge Distillation and Transfer Learning

“…Music works contain rich human emotions, and emotions play an indispensable role in the transmission of musical emotions and understanding and appreciation of music [1][2][3]. With the current development of Internet technology and artificial intelligence, the amount of digital music is growing rapidly.…”