Multimodal Emotion Classification in Naturalistic User Behavior

Walter, Steffen; Scherer, Stefan; Schels, Martin; Glodek, Michael; Hrabal, David; Schmidt, Manfred; Böck, Ronald; Limbrecht, Kerstin; Traue, Harald C.; Schwenker, Friedhelm

doi:10.1007/978-3-642-21616-9_68

Cited by 45 publications

(25 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Analysis of human emotions and processing recorded data, for instance the speech, facial expressions, hand gestures, body movements, etc. is a multidisciplinary field that has been emerging as a rich area of research in recent times [5,11,20,24,21,27]. In this paper multiple classifier systems for the classification of audio-visual features have been investigated, the numerical evaluation of the proposed emotion recognition systems has been carried out on the data sets of the AVEC challenge [23].…”

Section: Introductionmentioning

confidence: 99%

Multiple Classifier Systems for the Classification of Audio-Visual Emotional States

Glodek

Tschechne

Layher

et al. 2011

Lecture Notes in Computer Science

Self Cite

100

View full text Add to dashboard Cite

Abstract. Research activities in the field of human-computer interaction increasingly addressed the aspect of integrating some type of emotional intelligence. Human emotions are expressed through different modalities such as speech, facial expressions, hand or body gestures, and therefore the classification of human emotions should be considered as a multimodal pattern recognition problem. The aim of our paper is to investigate multiple classifier systems utilizing audio and visual features to classify human emotional states. For that a variety of features have been derived. From the audio signal the fundamental frequency, LPCand MFCC coefficients, and RASTA-PLP have been used. In addition to that two types of visual features have been computed, namely form and motion features of intermediate complexity. The numerical evaluation has been performed on the four emotional labels Arousal, Expectancy, Power, Valence as defined in the AVEC data set. As classifier architectures multiple classifier systems are applied, these have been proven to be accurate and robust against missing and noisy data.

show abstract

Section: Introductionmentioning

confidence: 99%

Multiple Classifier Systems for the Classification of Audio-Visual Emotional States

Glodek

Tschechne

Layher

et al. 2011

Lecture Notes in Computer Science

Self Cite

100

View full text Add to dashboard Cite

show abstract

“…This is unfortunately not the case in the presented examples with HCI: there seems to be a quick accustoming of the test subjects in the different scenarios (Rösner et al, 2012;Walter et al, 2013a). For example, playing a game twice in a row with the same types of feedback without any impact on the personal life of the subject will not render eligible results (Walter et al, 2011). It turns out that different levels of engagement can play an important role and stimuli that rely on surprise effects can justifiably be used only once.…”

Section: Intraindividual Experimentsmentioning

confidence: 91%

“…Similar effects occur when the problem of finding a ground truth is addressed by defining blocks of similar feedback from the system or the experimenter, grouped by the desired (and induced) user state (Walter et al, 2011(Walter et al, , 2013aRösner et al, 2012). This is often accompanied by a fixed order of the individual blocks as it may be infeasible to direct user states arbitrarily.…”

Section: Time-dependent Effectsmentioning

confidence: 94%

“…As the emotional stimulation has to occur in controllable settings, only relatively artificial scenarios are feasible that can be worked off during one short session of recording. This limits the narrative to very basic issues (McKeown et al, 2010;Walter et al, 2011;Rösner et al, 2012). In an experiment designed like that, the test subject has normally no real intrinsic interest in the task, other than to acquit oneself well in the experiment and to satisfy the experimenter.…”

Section: Involvement Of Test Subjectsmentioning

confidence: 99%

See 1 more Smart Citation

The Influence of Annotation, Corpus Design, and Evaluation on the Outcome of Automatic Classification of Human Emotions

2016

Self Cite

View full text Add to dashboard Cite

The integration of emotions into human-computer interaction applications promises a more natural dialog between the user and the technical system operators. In order to construct such machinery, continuous measuring of the affective state of the user becomes essential. While basic research that is aimed to capture and classify affective signals has progressed, many issues are still prevailing that hinder easy integration of affective signals into human-computer interaction. In this paper, we identify and investigate pitfalls in three steps of the work-flow of affective classification studies. It starts with the process of collecting affective data for the purpose of training suitable classifiers. Emotional data have to be created in which the target emotions are present. Therefore, human participants have to be stimulated suitably. We discuss the nature of these stimuli, their relevance to human-computer interaction, and the repeatability of the data recording setting. Second, aspects of annotation procedures are investigated, which include the variances of individual raters, annotation delay, the impact of the used annotation tool, and how individual ratings are combined to a unified label. Finally, the evaluation protocol is examined, which includes, among others, the impact of the performance measure on the accuracy of a classification model. We hereby focus especially on the evaluation of classifier outputs against continuously annotated dimensions. Together with the discussed problems and pitfalls and the ways how they affect the outcome, we provide solutions and alternatives to overcome these issues. As the final part of the paper, we sketch a recording scenario and a set of supporting technologies that can contribute to solve many of the issues mentioned above.

show abstract

“…There have been efforts to make machines social and emotionally aware [23]. There are methods to understand sentiments in human-computer dialogues [18], in naturalistic user behavior [24] and even in handwriting [26]. However, we are not aware of any work that estimates the underlying, experienced emotions in text conversations.…”

Section: Related Workmentioning

confidence: 99%

Exploring the Dynamics of Relationships Between Expressed and Experienced Emotions

Srinivasan

Chander

Dam

2017

Intelligent Human Computer Interaction

View full text Add to dashboard Cite

Abstract. Conversational user interfaces (CUIs) are rapidly evolving towards being ubiquitous as human-machine interfaces. Often, CUI backends are powered by a combination of human and machine intelligence, to address queries efficiently. Depending on the type of conversation issue, human-to-human conversations in CUIs (i.e. a human end-user conversing with the human in the CUI backend) could involve varying amounts of emotional content. While some of these emotions could be expressed through the conversation, others are experienced internally within the individual. Understanding the relationship between these two emotion modalities in the end-user could help to analyze and address the conversation issue better. Towards this, we propose an emotion analytic metric that can estimate experienced emotions based on its knowledge about expressed emotions in a user. Our findings point to the possibility of augmenting CUIs with an algorithmically guided emotional sense, which would help in having more effective conversations with end-users.

show abstract

Multimodal Emotion Classification in Naturalistic User Behavior

Cited by 45 publications

References 15 publications

Multiple Classifier Systems for the Classification of Audio-Visual Emotional States

Multiple Classifier Systems for the Classification of Audio-Visual Emotional States

The Influence of Annotation, Corpus Design, and Evaluation on the Outcome of Automatic Classification of Human Emotions

Exploring the Dynamics of Relationships Between Expressed and Experienced Emotions

Contact Info

Product

Resources

About