MixedEmotions: An Open-Source Toolbox for Multimodal Emotion Analysis

Abstract

Recently, there is an increasing tendency to embed functionalities for recognizing emotions from user generated media content in automated systems such as call-centre operations, recommendations and assistive technologies, providing richer and more informative user and content profiles. However, to date, adding these functionalities was a tedious, costly, and time consuming effort, requiring identification and integration of diverse tools with diverse interfaces as required by the use case at hand. The MixedEmotions Toolbox leverages the need for such functionalities by providing tools for text, audio, video, and linked data processing within an easily integrable plug-and-play platform. These functionalities include: (i) for text processing: emotion and sentiment recognition, (ii) for audio processing: emotion, age, and gender recognition, (iii) for video processing: face detection and tracking, emotion recognition, facial landmark localization, head pose estimation, face alignment, and body pose estimation, and (iv) for linked data: knowledge graph integration. Moreover, the MixedEmotions Toolbox is open-source and free. In this article, we present this toolbox in the context of the existing landscape, and provide a range of detailed benchmarks on standard test-beds showing its state-of-the-art performance. Furthermore, three real-world use-cases show its effectiveness, namely emotion-driven smart TV, call center monitoring, and brand reputation analysis.

E-ffective: A Visual Analytic System for Exploring the Emotion and Effectiveness of Inspirational Speeches

“…Vocal emotional data: With the timestamps of sentences, we split the audio into sentence-level clips and applied an open-source toolbox for multimodal emotion analysis developed by Buitelaar et al [6] to obtain the vocal valence and arousal data of the clips.…”

Suggestion Mining from Opinionated Text of Big Social Media Data

“…Prior approaches to suggestion mining focused on rules for linguistic and supervised machine learning through features that are manually identified. The key supervised learning algorithms used in these studies were the hidden Markov model, the conditional random field (CRF) [9], factorization machines [4], and SVM [2]. Further, these studies used training datasets that had less than 8,000 sentences and an exceedingly imbalanced distribution of classes.…”

“…The expression of sentences in available online text can be related to sentiments and emotions [1], and generally referred to as opinions, recommendations, instructions, advice, and tips for others regarding any entity. Such opinions can be collectively termed as suggestions [2].…”

“…Suggestion mining is an approach that largely emphasizes analyzing and identifying sentences to explore explicitly the suggestions they contain [2]. Identifying opinions about products and services that are discussed on social media is useful to organizations' management and to consumers.…”

Using R libraries to facilitate sensitivity analysis and to calibrate system dynamics models

“…Modern organizations now collect significant amounts of data, and for this to be of value the data must be analyzed to extract insights, which can then inform decision making (Kelleher et al ., ). Data analytics and machine learning have been deployed as a decision support approach in many fields, including: providing effective tools to improve outcomes and reduce costs in health care (Nevin, ); use in public health for infectious and non‐communicable diseases (Ginsberg et al ., ; Kristoufek et al ., ; Sinnenberg et al ., ); forecasting energy demand, wind generation and carbon dioxide emissions (Bai et al ., ; Mason et al ., ); and advances in natural language processing using techniques such as deep learning (Buitelaar et al ., ).…”