Deep Multimodal Learning for Affective Analysis and Retrieval

Pang, Lei; Zhu, Shiai; Ngo, Chong-Wah

doi:10.1109/tmm.2015.2482228

Cited by 132 publications

(69 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In particular, because of the complicated and unstructured nature of user-generated videos and the sparsity of video frames that express the emotion content, it is often hard to understand emotions conveyed in user-generated videos. To address this challenging problem, multi-modal fusion and knowledge transfer approaches have been proposed in recent works [28,45,49,46]. In this paper, we show that our FFCSN model can be easily extended to emotion recognition from user-generated videos, with state-of-the-art results achieved.…”

Section: Related Workmentioning

confidence: 78%

“…The YouTube-8 dataset [17] is used for performance evaluation. This dataset consists of 1,101 videos (downloaded from YouTube) annotated with 8 basic emotions: anger, an-Model multimodal ACC [17] visual+acoustic+attribute 46.1 [28] visual+acoustic+attribute 51.1 [49] visual+attribute 52.5 [46] visual+acoustic 52.6 [45] visual+acoustic 52.6 Ours visual 57.8 Table 3. Comparative results (%) of video-based emotion recognition on the YouTube-8 dataset.…”

Section: Dataset and Settingmentioning

confidence: 99%

See 1 more Smart Citation

Face-Focused Cross-Stream Network for Deception Detection in Videos

Ding

Zhao

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Automated deception detection (ADD) from real-life videos is a challenging task. It specifically needs to address two problems: (1) Both face and body contain useful cues regarding whether a subject is deceptive. How to effectively fuse the two is thus key to the effectiveness of an ADD model. (2) Real-life deceptive samples are hard to collect; learning with limited training data thus challenges most deep learning based ADD models. In this work, both problems are addressed. Specifically, for face-body multimodal learning, a novel face-focused cross-stream network (FFCSN) is proposed. It differs significantly from the popular two-stream networks in that: (a) face detection is added into the spatial stream to capture the facial expressions explicitly, and (b) correlation learning is performed across the spatial and temporal streams for joint deep feature learning across both face and body. To address the training data scarcity problem, our FFCSN model is trained with both meta learning and adversarial learning. Extensive experiments show that our FFCSN model achieves state-of-the-art results. Further, the proposed FFCSN model as well as its robust training strategy are shown to be generally applicable to other human-centric video analysis tasks such as emotion recognition from user-generated videos.

show abstract

Section: Related Workmentioning

confidence: 78%

Section: Dataset and Settingmentioning

confidence: 99%

Face-Focused Cross-Stream Network for Deception Detection in Videos

Ding

Zhao

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…In particular, understanding the sentiment in visual media content (i.e., images, videos) has attracted increasing research attention. Potential use of approaches developed for visual sentiment analysis is broad, including affective image retrieval [3], aesthetic quality categorization [4], opinion mining [5], comment assistant [6], etc.…”

Section: Introductionmentioning

confidence: 99%

Visual Sentiment Prediction Based on Automatic Discovery of Affective Regions

Yang

She

Sun

et al. 2018

IEEE Trans. Multimedia

151

View full text Add to dashboard Cite

Abstract-Automatic assessment of sentiment from visual content has gained considerable attention with the increasing tendency of expressing opinions via images and videos online. This paper investigates the problem of visual sentiment analysis, which involves a high-level abstraction in the recognition process. While most of the current methods focus on improving holistic representations, we aim to utilize the local information, which is inspired by the observation that both the whole image and local regions convey significant sentiment information. We propose a framework to leverage affective regions, where we first use an off-the-shelf objectness tool to generate the candidates, and employ a candidate selection method to remove redundant and noisy proposals. Then a convolutional neural network (CNN) is connected with each candidate to compute the sentiment scores, and the affective regions are automatically discovered, taking the objectness score as well as the sentiment score into consideration. Finally, the CNN outputs from local regions are aggregated with the whole images to produce the final predictions. Our framework only requires image-level labels, thereby significantly reducing the annotation burden otherwise required for training. This is especially important for sentiment analysis as sentiment can be abstract, and labeling affective regions is too subjective and labor-consuming. Extensive experiments show that the proposed algorithm outperforms the state-of-the-art approaches on eight popular benchmark datasets.

show abstract

“…Earlier work on emotional understanding of multimedia used hand crafted features from different modalities that are fused at feature or decision levels [8,15,46,117,126]. The more recent work mainly use deep learning models [55,93].…”

Section: Affective Computing Of Multimodal Datamentioning

confidence: 99%

“…Pang et al [93] used Deep Boltzmann Machine (DBM) to learn a joint representation across text, vision, and audio to recognize expected emotions from social media videos. Each modality is separately encoded with stacking multiple Restricted Boltzmann Machines (RBM) and pathways are merged to a joint representation layer.…”

Section: Affective Computing Of Multimodal Datamentioning

confidence: 99%

Affective Computing for Large-scale Heterogeneous Multimedia Data

Zhao

Wang

Soleymani

et al. 2019

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

The wide popularity of digital photography and social networks has generated a rapidly growing volume of multimedia data (i.e., image, music, and video), resulting in a great demand for managing, retrieving, and understanding these data. Affective computing (AC) of these data can help to understand human behaviors and enable wide applications. In this article, we survey the state-of-the-art AC technologies comprehensively for large-scale heterogeneous multimedia data. We begin this survey by introducing the typical emotion representation models from psychology that are widely employed in AC. We briefly describe the available datasets for evaluating AC algorithms. We then summarize and compare the representative methods on AC of different multimedia types, i.e., images, music, videos, and multimodal data, with the focus on both handcrafted features-based methods and deep learning methods. Finally, we discuss some challenges and future directions for multimedia affective computing.

show abstract

Deep Multimodal Learning for Affective Analysis and Retrieval

Cited by 132 publications

References 25 publications

Face-Focused Cross-Stream Network for Deception Detection in Videos

Face-Focused Cross-Stream Network for Deception Detection in Videos

Visual Sentiment Prediction Based on Automatic Discovery of Affective Regions

Affective Computing for Large-scale Heterogeneous Multimedia Data

Contact Info

Product

Resources

About