Deep Transfer Learning for Image Emotion Analysis: Reducing Marginal and Joint Distribution Discrepancies Together

He, Yuwei; Ding, Guiguang

doi:10.1007/s11063-019-10035-7

“…Based on CycleGAN, Zhao et al enforced semantic consistency when adapting the dominant emotions without requiring aligned image pairs [36,37]. He and Ding proposed a discrepancy-based domain adaptation method [154]. Both marginal and joint domain distribution discrepancies at fully-connected layers are reduced by minimizing the joint maximum mean discrepancy.…”

Section: Learning From Noisy Data or Few Labelsmentioning

confidence: 99%

Affective Image Content Analysis: Two Decades Review and New Perspectives

Zhao

¹

,

Yao

²

,

Yang

³

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges -the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.

show abstract

“…The area of emotion anal-ysis also received attention from the computer vision community. A common approach is to use transfer learning from general image classifiers (He and Ding, 2019) or the analysis of facial emotion expressions, with features of muscle movement (De Silva et al, 1997) or deep learning (Li and Deng, 2020). Dellagiacoma et al (2011) use texture and color features to analyze social media content.…”

Section: Related Workmentioning

confidence: 99%

On the Complementarity of Images and Text for the Expression of Emotions in Social Media

Khlyzova¹,

Silberer²,

Klinger³

2022

Preprint

0

View full text Add to dashboard Cite

Authors of posts in social media communicate their emotions and what causes them with text and images. While there is work on emotion and stimulus detection for each modality separately, it is yet unknown if the modalities contain complementary emotion information in social media. We aim at filling this research gap and contribute a novel, annotated corpus of English multimodal Reddit posts. On this resource, we develop models to automatically detect the relation between image and text, an emotion stimulus category and the emotion class. We evaluate if these tasks require both modalities and find for the imagetext relations, that text alone is sufficient for most categories (complementary, illustrative, opposing): the information in the text allows to predict if an image is required for emotion understanding. The emotions of anger and sadness are best predicted with a multimodal model, while text alone is sufficient for disgust, joy, and surprise. Stimuli depicted by objects, animals, food, or a person are best predicted by image-only models, while multimodal models are most effective on art, events, memes, places, or screenshots.

show abstract

“…The area of emotion analysis also received attention from the computer vi-sion community. A common approach is to use transfer learning from general image classifiers (He and Ding, 2019) or the analysis of facial emotion expressions, with features of muscle movement (De Silva et al, 1997) or deep learning (Li and Deng, 2020). Dellagiacoma et al (2011) use texture and color features to analyze social media content.…”

Section: Related Workmentioning

confidence: 99%

On the Complementarity of Images and Text for the Expression of Emotions in Social Media

Khlyzova¹,

Silberer²,

Klinger³

2022

Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment &Amp; Social Media Analysis

3

0

View full text Add to dashboard Cite

Authors of posts in social media communicate their emotions and what causes them with text and images. While there is work on emotion and stimulus detection for each modality separately, it is yet unknown if the modalities contain complementary emotion information in social media. We aim at filling this research gap and contribute a novel, annotated corpus of English multimodal Reddit posts. On this resource, we develop models to automatically detect the relation between image and text, an emotion stimulus category and the emotion class. We evaluate if these tasks require both modalities and find for the imagetext relations, that text alone is sufficient for most categories (complementary, illustrative, opposing): the information in the text allows to predict if an image is required for emotion understanding. The emotions of anger and sadness are best predicted with a multimodal model, while text alone is sufficient for disgust, joy, and surprise. Stimuli depicted by objects, animals, food, or a person are best predicted by image-only models, while multimodal models are most effective on art, events, memes, places, or screenshots.My everyday joy is to see my adorable cat smiles. And I've just realized, my cat can "dance with music". Amazing!

show abstract

Deep Transfer Learning for Image Emotion Analysis: Reducing Marginal and Joint Distribution Discrepancies Together

Cited by 12 publications

References 28 publications

Affective Image Content Analysis: Two Decades Review and New Perspectives

Affective Image Content Analysis: Two Decades Review and New Perspectives

On the Complementarity of Images and Text for the Expression of Emotions in Social Media

On the Complementarity of Images and Text for the Expression of Emotions in Social Media

Contact Info

Product

Resources

About