Human Affect Recognition: Audio‐Based Methods

Abstract

Automatic human affect recognition aims to automatically predict affect‐related information from humans observed for a certain time span. Such computer assessment of human emotion is described for audio‐based methods. This includes first acoustic analysis with suited features and segmentation of the speech signal. Then follows linguistic analysis including speech‐to‐text processing and representation of the text in a feature space, as well as joint acoustic–linguistic processing. Subsequently, data collection is described including emotion representation and elicitation. Further, automatic recognition by suited machine learning algorithms is touched upon. Then, solutions to cope with data scarcity, such as active, semi‐supervised, and transfer learning or even zero‐resource recognition, are explained. Following this, evaluation is discussed and typical benchmark results are given alongside standards and software. Next, application examples are given, before concluding by current research challenges.

Modelling User Affect and Sentiment in Intelligent User Interfaces

“…In fact, they are expected to play a major role that will likely make 'that difference' in future intelligent User Interfaces (IUIs), as they bear the promise to lend them emotional intelligence: Interfaces that 'know' and can react appropriately to, e. g., the satisfaction or anger of their users can lead us away from the often prevailing connotation of 'cold' and 'mechanical' that the interfaces of the current generation are still partially faced with. The information is thereby increasingly accessed from multiple modalities [15,17] -both, in affect recognition and sentiment analysis -and 'in the wild' [33,9], thanks to the availability of increasingly large and realistic resources [24] and improved algorithms [21] including deep learning [26] and long-short-term memory architectures [14] and weakly supervised learning methods [25]. Besides a certain focus on analysis in research to date, a system that emulates emotion is likely to be perceived as even more 'emotionally intelligent' given that it manages to overcome the uncanny valley.…”