Deep multimodal fusion for persuasiveness prediction

Nojavanasghari, Behnaz; Gopinath, Deepak; Koushik, Jayanth; Baltrušaitis, Tadas; Morency, Louis–Philippe

doi:10.1145/2993148.2993176

Cited by 165 publications

(89 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This baseline is trained on the concatenated multimodal features for classification or regression task (Pérez-Rosas et al, 2013), (Park et al, 2014a), (Zadeh et al, 2016b). Deep Fusion The Deep Fusion model (DF) (Nojavanasghari et al, 2016) trains one deep neural model for each modality and then combine the output of each modality network with a joint neural network. Tensor Fusion Network The Tensor Fusion Network (TFN) explicitly models view-specific and cross-view dynamics by creating a multi-dimensional tensor that captures uni-modal, bimodal and trimodal interactions across three modalities.…”

Section: Baseline Modelsmentioning

confidence: 99%

Efficient Low-rank Multimodal Fusion With Modality-Specific Factors

Liu

Shen

Lakshminarasimhan³

et al. 2018

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

440

217

View full text Add to dashboard Cite

Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exploited the expressiveness of tensors for multimodal representation. However, these methods often suffer from exponential increase in dimensions and in computational complexity introduced by transformation of input into tensor. In this paper, we propose the Lowrank Multimodal Fusion method, which performs multimodal fusion using low-rank tensors to improve efficiency. We evaluate our model on three different tasks: multimodal sentiment analysis, speaker trait analysis, and emotion recognition. Our model achieves competitive results on all these tasks while drastically reducing computational complexity. Additional experiments also show that our model can perform robustly for a wide range of low-rank settings, and is indeed much more efficient in both training and inference compared to other methods that utilize tensor representations.

show abstract

Section: Baseline Modelsmentioning

confidence: 99%

Efficient Low-rank Multimodal Fusion With Modality-Specific Factors

Liu

Shen

Lakshminarasimhan³

et al. 2018

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

440

217

View full text Add to dashboard Cite

show abstract

“…We used 40% of the data for training, 30% for validation and 30% for testing. We used majority voting, naive Bayes, radial basis function kernel SVM [36], binary logistic regression [37] and a deep neural network [38] as our baseline models. We used the validation set for selecting the hyper parameters of our models.…”

Section: Experimental Methodologymentioning

confidence: 99%

Hand2Face: Automatic synthesis and recognition of hand over face occlusions

Nojavanasghari

Hughes

Baltrušaitis

et al. 2017

2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)

Self Cite

View full text Add to dashboard Cite

A person's face discloses important information about their affective state. Although there has been extensive research on recognition of facial expressions, the performance of existing approaches is challenged by facial occlusions. Facial occlusions are often treated as noise and discarded in recognition of affective states. However, hand over face occlusions can provide additional information for recognition of some affective states such as curiosity, frustration and boredom. One of the reasons that this problem has not gained attention is the lack of naturalistic occluded faces that contain hand over face occlusions as well as other types of occlusions. Traditional approaches for obtaining affective data are time demanding and expensive, which limits researchers in affective computing to work on small datasets. This limitation affects the generalizability of models and deprives researchers from taking advantage of recent advances in deep learning that have shown great success in many fields but require large volumes of data. In this paper, we first introduce a novel framework for synthesizing naturalistic facial occlusions from an initial dataset of non-occluded faces and separate images of hands, reducing the costly process of data collection and annotation. We then propose a model for facial occlusion type recognition to differentiate between hand over face occlusions and other types of occlusions such as scarves, hair, glasses and objects. Finally, we present a model to localize hand over face occlusions and identify the occluded regions of the face.

show abstract

“…Vectorized features from different sources of knowledge can be combined in deep learning using a simple process, such as concatenation or weighted sum, which often has just a few or even number of parameters involved because, the joint training of the deep models will change the layers for high-level extractions of features to compensate for the process needed. Concatenation may be used to combine either low input [6], [7] characteristics or high feature derived from the pre-trained models [8], [9]. Proposed model uses the first technique that is simple operation-based fusion where the vectorized features from images are integrated using concatenation.…”

Section: Related Workmentioning

confidence: 99%

Image Captioning with Complementary Visual and Textual Cues

Venkatesan¹,

Sond²

2020

Preprint

View full text Add to dashboard Cite

Describing an image with natural sentence without human involvement can be achieved using Deep Neural network, it requires knowledge of both image processing and Natural language processing. Most of the existing works are based on single modality model with Encoder-Decoder architecture where input images are encoded using Convolution Neural Network (CNN) and caption is generated by Recurrent Neural Network (RNN). In this paper, we propose image captioning model with complementary visual and textual cues. Our model performs early fusion by combining encoded image features from different CNNs, along with combined textual features from different word embedding techniques. The fused inputs are passed to our language model Long Short-Term Memory (LSTM) which generate captions. The result shows that our model with additional complementary information outperforms existing single modality models.

show abstract

Deep multimodal fusion for persuasiveness prediction

Cited by 165 publications

References 19 publications

Efficient Low-rank Multimodal Fusion With Modality-Specific Factors

Efficient Low-rank Multimodal Fusion With Modality-Specific Factors

Hand2Face: Automatic synthesis and recognition of hand over face occlusions

Image Captioning with Complementary Visual and Textual Cues

Contact Info

Product

Resources

About