Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Abstract

Humour is a substantial element of human affect and cognition. Its automatic understanding can facilitate a more naturalistic human-device interaction and the humanisation of artificial intelligence. Current methods of humour detection are solely based on staged data making them inadequate for 'real-world' applications. We address this deficiency by introducing the novel Passau-Spontaneous Football Coach Humour (Passau-SFCH) dataset, comprising of about 11 hours of recordings. The Passau-SFCH dataset is annotated for the presence of humour and its dimensions (sentiment and direction) as proposed in Martin's Humor Style Questionnaire. We conduct a series of experiments, employing pretrained Transformers, convolutional neural networks, and expert-designed features. The performance of each modality (text, audio, video) for spontaneous humour recognition is analysed and their complementarity is investigated. Our findings suggest that for the automatic analysis of humour and its sentiment, facial expressions are most promising, while humour direction can be best modelled via text-based features. The results reveal considerable differences among various subjects, highlighting the individuality of humour usage and style. Further, we observe that a decision-level fusion yields the best recognition result. Finally, we make our code publicly available at https://www.github.com/EIHW/passau-sfch. The Passau-SFCH dataset is available upon request.

The MuSe 2022 Multimodal Sentiment Analysis Challenge

“…The MuSe-Humour Sub-Challenge is based on an extension of the Passau-SFCH dataset [18], aiming at the recognition of spontaneous humour in press conferences. A cross-cultural setting is put forward for this task, where participants train their model on German recordings, but have to predict humorous utterances in English data.…”

“…Different from the training data, the unseen test set consists of videos of English football press conferences, thus providing a cross-cultural evaluation setting. The Passau Spontaneous Football Coach Humour (Passau-SFCH) dataset [18] as featured in MuSe 2022 [17,38,63] is provided as the training set. For the test set, we extend Passau-SFCH with recordings of press conferences given by 7 different coaches from the English Premier League between September 2016 and September 2020.…”