Controlling for Confounders in Multimodal Emotion Classification via Adversarial Learning

Abstract

Various psychological factors affect how individuals express emotions. Yet, when we collect data intended for use in building emotion recognition systems, we often try to do so by creating paradigms that are designed just with a focus on eliciting emotional behavior. Algorithms trained with these types of data are unlikely to function outside of controlled environments because our emotions naturally change as a function of these other factors. In this work, we study how the multimodal expressions of emotion change when an individual is under varying levels of stress. We hypothesize that stress produces modulations that can hide the true underlying emotions of individuals and that we can make emotion recognition algorithms more generalizable by controlling for variations in stress. To this end, we use adversarial networks to decorrelate stress modulations from emotion representations. We study how stress alters acoustic and lexical emotional predictions, paying special attention to how modulations due to stress affect the transferability of learned emotion recognition models across domains. Our results show that stress is indeed encoded in trained emotion classifiers and that this encoding varies across levels of emotions and across the lexical and acoustic modalities. Our results also show that emotion recognition models that control for stress during training have better generalizability when applied to new domains, compared to models that do not control for stress during training. We conclude that is is necessary to consider the effect of extraneous psychological factors when building and testing emotion recognition models.

A Review on Methods and Applications in Multimodal Deep Learning

“…The experimental results showed the dominance of GET as compared to other models of image captioning. Experimental results on all listed MMID models are compiled and listed separately for each R-LSTM [39] RFNet [40] He [42] Feng [43] GET [44] Wang [45] FCN-LSTM [46] Bag-LSTM [47] Stack-VS [48] VSR [49] GLA [50] Up-Down [51] MAGAN [55] MGAN [56] R 0 5 10 15 20 25 30 35 40 RFNet [40] He [42] Feng [43] GET [44] Wang [45] FCN-LSTM [46] Bag-LSTM [47] Stack-VS [48] VSR [49] GLA [50] Up-Down [51] MAGAN [55] MGAN [56] S R-LSTM [39] RFNet [40] He [42] Feng [43] GET [44] Wang [45] FCN-LSTM [46] Bag-LSTM [47] Stack-VS [48] VSR [49] GLA [50] Up-Down R-LSTM [39] RFNet [40] He [42] Feng [43] GET [44] Wang [45] FCN-LSTM [46] Bag-LSTM…”

“…CRNN [38] R-LSTM [39] RFNet [40] He [42] Feng [43] GET [44] Wang [45] FCN-LSTM [46] Bag-LSTM [47] Stack-VS [48] VSR [49] GLA [50] Up-Down [51] MAGAN [55] MGAN [56] M…”

“…In this framework, audio and textual modalities are used for the detection of emotions. M. Jaiswal et al [46] analyzes the change of emotional expressions under various stress levels of an individual. The performance of this task is affected by the degree of lexical or acoustic features.…”

Recent Advances and Trends in Multimodal Deep Learning: A Review

“…In this framework, audio and textual modalities are used for the detection of emotions. M. Jaiswal et al [67] analyzes the change of emotional expressions under various stress levels of an individual. The performance of this task is affected by the degree of lexical or acoustic features.…”