Frustration recognition from speech during game interaction using wide residual networks

Song, Meishu; Mallol-Ragolta, Adria; Parada-Cabaleiro, Emilia; Yang, Zijiang; Liu, Shuo; Ren, Zhao; Zhao, Ziping; Schuller, Björn W.

doi:10.1016/j.vrih.2020.10.004

Cited by 11 publications

(2 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fig. 2), widely utilised in speech emotion recognition [35], were considered appropriate for the purposes of the present study. As a standard procedure in the field, the experiments were carried out in a speaker-independent manner, considering 43 speakers for training, 12 speakers for development (i.e., validation), and 12 speakers for test.…”

Section: Methodsmentioning

confidence: 99%

Supervised Contrastive Learning for Game-Play Frustration Detection from Speech

Song

Parada-Cabaleiro

Liu

et al. 2021

Universal Access in Human-Computer Interaction. Design Methods and User Experience

Self Cite

View full text Add to dashboard Cite

Frustration is a common response during game interactions, typically decreasing a user's engagement and leading to game failure. Artificially intelligent methods capable to automatically detect a user's level of frustration at an early stage are hence of great interest for game designers, since this would enable optimisation of a player's experience in real-time. Nevertheless, research in this context is still in its infancy, mainly relying on the use of pre-trained models and fine-tuning tailored to a specific dataset. Furthermore, this lack in research is due to the limited data available and to the ambiguous labelling of frustration, which leads to outcomes which are not generalisable in the real-world. Meanwhile, contrastive loss has been considered instead of the traditional cross-entropy loss in a variety of machine learning applications, showing to be more robust for system stability alternative in self-supervised learning. Following this trend, we hypothesise that using a supervised contrastive loss might overcome the limitations of the cross-entropy loss yielded by the labels' ambiguity. In fact, our experiments demonstrate that using the supervised contrastive method as a loss function, results improve for the automatic recognition (binary frustration vs nofrustration) of game-induced frustration from speech with an Unweighted Average Recall increase from 86.4 % to 89.9 %.

show abstract

Section: Methodsmentioning

confidence: 99%

Supervised Contrastive Learning for Game-Play Frustration Detection from Speech

Song

Parada-Cabaleiro

Liu

et al. 2021

Universal Access in Human-Computer Interaction. Design Methods and User Experience

Self Cite

View full text Add to dashboard Cite

show abstract

“…State-of-the-art deep learning algorithms outperformed classical approaches for call-center anger recognition (Deng et al, 2017). Wizzard-of-Oz setups are a successful method for emotion elicitation, and in Song et al (2021) this method was used to elicit frustration in participants playing a game and recognizing frustration using a range of speech features and deep learning architectures. While in research related to negative emotions the focus and annotation is often perform per person, in our work the focus is on the interaction between subjects and how it evolves.…”

Section: Related Workmentioning

confidence: 99%

A Cross-Corpus Speech-Based Analysis of Escalating Negative Interactions

Lefter

Baird

Stappen

et al. 2022

Front. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

The monitoring of an escalating negative interaction has several benefits, particularly in security, (mental) health, and group management. The speech signal is particularly suited to this, as aspects of escalation, including emotional arousal, are proven to easily be captured by the audio signal. A challenge of applying trained systems in real-life applications is their strong dependence on the training material and limited generalization abilities. For this reason, in this contribution, we perform an extensive analysis of three corpora in the Dutch language. All three corpora are high in escalation behavior content and are annotated on alternative dimensions related to escalation. A process of label mapping resulted in two possible ground truth estimations for the three datasets as low, medium, and high escalation levels. To observe class behavior and inter-corpus differences more closely, we perform acoustic analysis of the audio samples, finding that derived labels perform similarly across each corpus, with escalation interaction increasing in pitch (F0) and intensity (dB). We explore the suitability of different speech features, data augmentation, merging corpora for training, and testing on actor and non-actor speech through our experiments. We find that the extent to which merging corpora is successful depends greatly on the similarities between label definitions before label mapping. Finally, we see that the escalation recognition task can be performed in a cross-corpus setup with hand-crafted speech features, obtaining up to 63.8% unweighted average recall (UAR) at best for a cross-corpus analysis, an increase from the inter-corpus results of 59.4% UAR.

show abstract