Emotion classification: How does an automated system compare to Naive human coders?

Abstract

The fact that emotions play a vital role in social interactions, along with the demand for novel human-computer interaction applications, have led to the development of a number of automatic emotion classification systems. However, it is still debatable whether the performance of such systems can compare with human coders. To address this issue, in this study, we present a comprehensive comparison in a speech-based emotion classification task between 138 Amazon Mechanical Turk workers (Turkers) and a state-of-the-art automatic computer system. The comparison includes classifying speech utterances into six emotions (happy, neutral, sad, anger, disgust and fear), into three arousal classes (active, passive, and neutral), and into three valence classes (positive, negative, and neutral). The results show that the computer system outperforms the naive Turkers in almost all cases. Furthermore, the computer system can increase the classification accuracy by rejecting to classify utterances for which it is not confident, while the Turkers do not show a significantly higher classification accuracy on their confident utterances versus unconfident ones.

Enhanced multiclass SVM with thresholding fusion for speech-based emotion classification

“…As an example, the decision-level correct classification rate can be increased to 93% and 56% when half of the samples are rejected as unclassified for the aforementioned LDC dataset and UGA dataset, respectively. This can be contrasted with results from a human user study presented in Eskimez et al (2016), in which naive coders on Amazon Mechanical Turk were asked to classify the emotions in the LDC dataset. Results from this test show that naive human coders cannot improve their classification accuracy by rejecting samples where they are not confident in their decision.…”

“…However, it is not clear how well this system, which in many applications would replace human classification of the emotion, compares to a naive human coder performing the same emotion classification task. Therefore, in our work in Eskimez et al (2016), we asked Amazon Mechanical Turk workers (Turkers) to listen to speech samples from the LDC dataset of emotions and classify them into six categories. There were 138 unique Turkers that classified 7,270 audio samples, with individual Turkers classifying between 10 and 100 audio samples.…”

Emotion Detection from Speech and Text

“…However, this is a challenging task for an automatic system. In recent years, the great amount of multimedia information available due to the extensive use of the Internet and social media, along with new computational methodologies related to machine learning, have led to the scientific community to put a great effort in this area [1,2].…”

Tracking the Expression of Annoyance in Call Centers

“…In recent year, the machine learning community has paid increasing attention to model the emotional status based on parameters derived from the analysis of the voice, the language, the face, the gestures or the ECG [2]. But data-driven approaches need corpora of human spontaneous behavior annotated with emotional labels [2] [3], which is a challenging requirement mainly due to the subjectivity of emotion perception by humans [2] [4]. As a consequence much research is being carried out over corpora that cover simulated or induced emotional behavior [4] [5].…”

“…But data-driven approaches need corpora of human spontaneous behavior annotated with emotional labels [2] [3], which is a challenging requirement mainly due to the subjectivity of emotion perception by humans [2] [4]. As a consequence much research is being carried out over corpora that cover simulated or induced emotional behavior [4] [5]. It is worth mentioning that the selection of the situation where spontaneous emotions can be collected strongly depends on the goals of the research to be carried out.…”