Backchannel Detection and Agreement Estimation from Video with Transformer Networks

MultiMediate '23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions

“…As in MultiMediate '21 [37], our challenge is based on the MPIIGroupInteraction dataset [38,39]. This dataset has served as a basis for diverse tasks, including emergent leadership detection [35], eye contact detection [18,30,38], next speaker prediction [9], backchannel analysis [1,52], and body language detection [3]. The MPIIGroupInteraction corpus consists of 22 group discussions between three to four people, each lasting for 20 minutes [39].…”

“…With the multi-year MultiMediate challenge we contribute to realising the vision of autonomous artificial mediators by facilitating measurable advances on central conversational behaviour sensing and analysis tasks. The first iteration of the challenge in 2021 [37] has addressed eye contact detection and next speaker prediction while MultiMediate '22 has focused on backchannel analysis [1,36]. In two separate tracks, MultiMediate '23 addresses the recognition of complex bodily behaviours, as well as the estimation of a persons' engagement level.…”

“…We make all collected annotations, baseline implementations, and raw feature representations publicly available for further use, even beyond the scope of MultiMediate '23. 1…”