The USC CreativeIT database of multimodal dyadic interactions: from speech and full body motion capture to continuous emotional annotations

Metallinou, Angeliki; Yang, Zhaojun; Lee, Chi-Chun; Busso, Carlos; Carnicke, Sharon Marie; Narayanan, Shrikanth

doi:10.1007/s10579-015-9300-0

Cited by 54 publications

(33 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More specifically, several problems manifested in absolute ratings such as sensitivity to the range, anchor point and sequential effects (Parthasarathy et al, 2016) among different annotators would cause noise in the first-order delta. Interrater agreement refers to agreement on categorical or numeric annotations among annotators, which is normally measured by Cronbach's α (McKeown et al, 2012;Ringeval et al, 2013;Metallinou et al, 2015).…”

Section: Delta Emotion Ground Truthmentioning

confidence: 99%

“…With growing awareness of this, there has been an increasing number of groups considering the time course of emotions by employing continuously annotated corpora (Gunes and Schuller, 2013). Examples of this kind of corpora are SEMAINE , CreativeIT (Metallinou et al, 2015), RECOLA (Ringeval et al, 2013) and Belfast Naturalistic Database (Sneddon et al, 2012), where emotional ratings (e.g., arousal and valence) are evaluated continuously using real-time annotation tools such as Feeltrace (Cowie and Douglas-Cowie, 2000), Gtrace (Cowie et al, 2013), and ANNEMO (Ringeval et al, 2013), based on audio and video signals. Based on continuous annotation, a number of systems have been built with the intention of predicting the ratings at a fine temporal granularity, for example, the Audio-visual Emotion Challenge (AVEC) (Schuller et al, 2011;Ringeval et al, 2015b), but overall performances are not always satisfactory.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Prediction of Emotion Change From Speech

Huang

Epps

2018

Front. ICT

View full text Add to dashboard Cite

The fact that emotions are dynamic in nature and evolve across time has been explored relatively less often in automatic emotion recognition systems to date. Although withinutterance information about emotion changes recently has received some attention, there remain open questions unresolved, such as how to approach delta emotion ground truth, how to predict the extent of emotion change from speech, and how well change can be predicted relative to absolute emotion ratings. In this article, we investigate speech-based automatic systems for continuous prediction of the extent of emotion changes in arousal/valence. We propose the use of regression (smoothed) deltas as ground truth for emotion change, which yielded considerably higher inter-rater reliability than first-order deltas, a commonly used approach in previous research, and represent a more appropriate approach to derive annotations for emotion change research, findings which are applicable beyond speech-based systems. In addition, the first system design for continuous emotion change prediction from speech is explored. Experimental results under the Output-Associative Relevance Vector Machine framework interestingly show that changes in emotion ratings may be better predicted than absolute emotion ratings on the RECOLA database, achieving 0.74 vs. 0.71 for arousal and 0.41 vs. 0.37 for valence in concordance correlation coefficients. However, further work is needed to achieve effective emotion change prediction performances on the SEMAINE database, due to the large number of non-change frames in the absolute emotion ratings.

show abstract

Section: Delta Emotion Ground Truthmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Prediction of Emotion Change From Speech

Huang

Epps

2018

Front. ICT

View full text Add to dashboard Cite

show abstract

“…LGBPTOP has been used as baseline feature in automatic affect recognition challenge [3] [11]. Video geometric features include identifying landmarks on the face [11] or shoulder [12] or the whole body [13].…”

Section: Related Workmentioning

confidence: 99%

Linear and Non-Linear Multimodal Fusion for Continuous Affect Estimation In-the-Wild

Gaus

Meng

2018

2018 13th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2018)

View full text Add to dashboard Cite

Automatic continuous affect recognition from multiple modality in the wild is arguably one of the most challenging research areas in affective computing. In addressing this regression problem, the advantages of the each modality, such as audio, video and text, have been frequently explored but in an isolated way. Little attention has been paid so far to quantify the relationship within these modalities. Motivated to leverage the individual advantages of each modality, this study investigates behavioral modeling of continuous affect estimation, in multimodal fusion approaches, using Linear Regression, Exponent Weighted Decision Fusion and Multi-Gene Genetic Programming. The capabilities of each fusion approach are illustrated by applying it to the formulation of affect estimation generated from multiple modality using classical Support Vector Regression. The proposed fusion methods were applied in the public Sentiment Analysis in the Wild (SEWA) multimodal dataset and the experimental results indicate that employing proper fusion can deliver a significant performance improvement for all affect estimation. The results further show that the proposed systems is competitive or outperform the other state-of-the-art approaches.

show abstract

“…An interactive emotional dyadic motion capture, named the USC IEMOCAP database, is presented in [19], which is a multimodal and multi-speaker database of improvised and scripted dyadic interactions. The USC Cre-ativeIT database contains full-body motion capture information in the context of expressive theatrical improvisations [9,20]. The database is annotated using the valence, activation and dominance attributes, as well as the theater performance ratings such as interest and naturalness.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Through social signals of agreement and disagreement in a communicative interaction participants can share convergent or divergent opinions, proposals, goals, attitudes and feelings. In recent literature common types of such social interaction are the group meeting scenarios [2][3][4][5], political debates [6][7][8], theatrical improvisations [9] and broadcast conversations [10,11].…”

Section: Introductionmentioning

confidence: 99%

Use of Agreement/Disagreement Classification in Dyadic Interactions for Continuous Emotion Recognition

Khaki

Erzin

2016

Interspeech 2016

View full text Add to dashboard Cite

Natural and affective handshakes of two participants define the course of dyadic interaction. Affective states of the participants are expected to be correlated with the nature or type of the dyadic interaction. In this study, we investigate relationship between affective attributes and nature of dyadic interaction. In this investigation we use the JESTKOD database, which consists of speech and full-body motion capture data recordings for dyadic interactions under agreement and disagreement scenarios. The dataset also has affective annotations in activation, valence and dominance (AVD) attributes. We pose the continuous affect recognition problem under agreement and disagreement scenarios of dyadic interactions. We define a statistical mapping using the support vector regression (SVR) from speech and motion modalities to affective attributes with and without the dyadic interaction type (DIT) information. We observe an improvement in estimation of the valence attribute when the DIT is available. Furthermore this improvement sustains even we estimate the DIT from the speech and motion modalities of the dyadic interaction.

show abstract

The USC CreativeIT database of multimodal dyadic interactions: from speech and full body motion capture to continuous emotional annotations

Cited by 54 publications

References 43 publications

Prediction of Emotion Change From Speech

Prediction of Emotion Change From Speech

Linear and Non-Linear Multimodal Fusion for Continuous Affect Estimation In-the-Wild

Use of Agreement/Disagreement Classification in Dyadic Interactions for Continuous Emotion Recognition

Contact Info

Product

Resources

About