MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception

Busso, Carlos; Parthasarathy, S.; Burmania, Alec; Abdelwahab, Mohammed; Sadoughi, Najmeh; Provost, Emily Mower

doi:10.1109/taffc.2016.2515617

Cited by 277 publications

(179 citation statements)

References 44 publications

Supporting

Mentioning

164

Contrasting

Order By: Relevance

“…The evaluations were conducted on Amazon's Mechanical Turk [22] ("MTurk"). MTurk, and crowd sourcing platforms more generally, have seen increasing use in the fields of emotion modeling [20], [21], [36], [51], [52], sentiment analysis [53], analysis of social behavior [54], transcription [55], [56], perceptual evaluation [57], [58], and assistive technology for the blind [59]. These platforms allow for the rapid collection of many evaluations from a large disparate user population [60].…”

Section: Discussionmentioning

confidence: 99%

“…We chose to embed the utterances in improvised scenarios rather than rely upon read speech (data recorded by an individual reading an utterance in different emotions) to enhance the naturalness of the expression (see [21] for a discussion of the naturalness of the utterances). For example, the sentence, "I am going shopping," was used to admonish a roommate who ate all the snacks in the house prior to a party (emotion: anger) and as an expression of celebration following a promotion at work (emotion: happiness).…”

Section: Original Recordingmentioning

confidence: 99%

“…Each utterance was performed in a dyadic interaction with a "scene partner" acting as a friend to whom the actor was recounting his/her experiences. Please see [21] for more detail regarding the OAV recording process.…”

Section: Original Recordingmentioning

confidence: 99%

“…We manipulate these stimuli to create a new set of stimuli with mismatched emotion content. This paradigm allows us to obtain the fixed lexical content needed to synthesize emotionally mismatched utterances while allowing for enhanced naturalness compared to read speech collection paradigms [21]. We evaluate the emotion content using Amazon Mechanical Turk [22], used in our previous work [20], and model the resulting perceptual patterns.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

UMEME: University of Michigan Emotional McGurk Effect Data Set

Provost

Shangguan

Busso

2015

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

Emotion is central to communication; it colors our interpretation of events and social interactions. Emotion expression is generally multimodal, modulating our facial movement, vocal behavior, and body gestures. The method through which this multimodal information is integrated and perceived is not well understood. This knowledge has implications for the design of multimodal classification algorithms, affective interfaces, and even mental health assessment. We present a novel dataset designed to support research into the emotion perception process, the University of Michigan Emotional McGurk Effect Dataset (UMEME). UMEME has a critical feature that differentiates it from currently existing datasets; it contains not only emotionally congruent stimuli (emotionally matched faces and voices), but also emotionally incongruent stimuli (emotionally mismatched faces and voices). The inclusion of emotionally complex and dynamic stimuli provides an opportunity to study how individuals make assessments of emotion content in the presence of emotional incongruence, or emotional noise. We describe the collection, annotation, and statistical properties of the data and present evidence illustrating how audio and video interact to result in specific types of emotion perception. The results demonstrate that there exist consistent patterns underlying emotion evaluation, even given incongruence, positioning UMEME as an important new tool for understanding emotion perception.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Original Recordingmentioning

confidence: 99%

Section: Original Recordingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

UMEME: University of Michigan Emotional McGurk Effect Data Set

Provost

Shangguan

Busso

2015

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Busso et al provided an emotion dataset from speech and gesture, including facial expressions and hand movements [6]. The improved version of that dataset provided an affective database with audiovisual information which promoting naturalness within the (acted) recording [7].…”

Section: Related Workmentioning

confidence: 99%

Multitask Learning and Multistage Fusion for Dimensional Audiovisual Emotion Recognition

Atmaja

Akagi

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Due to its ability to accurately predict emotional state using multimodal features, audiovisual emotion recognition has recently gained more interest from researchers. This paper proposes two methods to predict emotional attributes from audio and visual data using a multitask learning and a fusion strategy. First, multitask learning is employed by adjusting three parameters for each attribute to improve the recognition rate. Second, a multistage fusion is proposed to combine results from various modalities' final prediction. Our approach used multitask learning, employed at unimodal and early fusion methods, shows improvement over single-task learning with an average CCC score of 0.431 compared to 0.297. A multistage method, employed at the late fusion approach, significantly improved the agreement score between true and predicted values on the development set of data (from [0.537, 0.565, 0.083] to [0.68, 0.656, 0.443]) for arousal, valence, and liking.

show abstract

EmoBone: A Multinational Audio Dataset of Emotional Bone Conducted Speech

Hosain,

Sugiura,

Rahman

et al. 2024

IEEJ Transactions Elec Engng

View full text Add to dashboard Cite

This paper introduces EmoBone, a comprehensive audio‐only emotional bone‐conducted speech dataset featuring speakers from various countries. The dataset comprises speeches from 28 individuals representing 10 different nations, with each participant delivering 10 sentences designed to evoke distinct emotions. In addition to an air‐conducted microphone, the recordings utilized bone conduction technology, transmitting sound directly to the speakers' inner ears, ensuring high‐quality emotional speech recordings. To assess the validity of the dataset, 80 university students from Bangladesh listened to the recordings and successfully identified the expressed emotions with an accuracy exceeding 76%. Statistical methods were also employed to evaluate the reliability of the dataset, revealing a high level of agreement among raters. EmoBone, with a cumulative duration surpassing 19 h and 15 680 unique utterances, stands as the most extensive emotional speech dataset available. This makes it a valuable tool for studying how emotional speech varies across cultures. Furthermore, due to its utilization of bone conduction technology, EmoBone facilitates the study of acoustic features in emotional speech from diverse dimensions. The data that supports the findings of this study is available upon reasonable request. © 2024 Institute of Electrical Engineer of Japan and Wiley Periodicals LLC.

show abstract

MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception

Cited by 277 publications

References 44 publications

UMEME: University of Michigan Emotional McGurk Effect Data Set

UMEME: University of Michigan Emotional McGurk Effect Data Set

Multitask Learning and Multistage Fusion for Dimensional Audiovisual Emotion Recognition

EmoBone: A Multinational Audio Dataset of Emotional Bone Conducted Speech

Contact Info

Product

Resources

About