Deep learning based multimodal emotion recognition using model-level fusion of audio–visual modalities

Middya, Asif Iqbal; Nag, B.; Roy, Sarbani

doi:10.1016/j.knosys.2022.108580

Cited by 97 publications

(28 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, it was observed that in order to gain a higher confidence level in the recognition of emotions, the combination of audio and video data is needed. This is also in line with the studies reported in [4,5,[54][55][56][57][58][59][60]. For some emotions (such as angry) audio cues play a more important role, whereas for some other emotions (such as happy) video cues are more important and the combination of the both cues improves the confidence in emotion recognition.…”

Section: Neutral (Non-expressive)supporting

confidence: 91%

Subjective Evaluation of Basic Emotions from Audio–Visual Data

Kadiri

Alku

2022

Sensors

View full text Add to dashboard Cite

Understanding of the perception of emotions or affective states in humans is important to develop emotion-aware systems that work in realistic scenarios. In this paper, the perception of emotions in naturalistic human interaction (audio–visual data) is studied using perceptual evaluation. For this purpose, a naturalistic audio–visual emotion database collected from TV broadcasts such as soap-operas and movies, called the IIIT-H Audio–Visual Emotion (IIIT-H AVE) database, is used. The database consists of audio-alone, video-alone, and audio–visual data in English. Using data of all three modes, perceptual tests are conducted for four basic emotions (angry, happy, neutral, and sad) based on category labeling and for two dimensions, namely arousal (active or passive) and valence (positive or negative), based on dimensional labeling. The results indicated that the participants’ perception of emotions was remarkably different between the audio-alone, video-alone, and audio–video data. This finding emphasizes the importance of emotion-specific features compared to commonly used features in the development of emotion-aware systems.

show abstract

Section: Neutral (Non-expressive)supporting

confidence: 91%

Subjective Evaluation of Basic Emotions from Audio–Visual Data

Kadiri

Alku

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…However, although these measures have eased the constraints on credit supply, they have not yet achieved ideal results in improving the credit availability to herders. The reason is that the credit availability of herdsmen is not only related to the credit rationing of financial institutions but also influenced by herdsmen's characteristics, especially herdsmen's psychological characteristics [ 22 ]. Figure 4 reveals the effect of herdsman's emotional and psychological characteristics on herdsman's credit demand.…”

Section: Methodsmentioning

confidence: 99%

Analysis of Herdsmen’s Credit Behavior under the Identification Scheme of Emotional and Psychological State

Zhang

Huang

Liyang

2022

Occupational Therapy International

View full text Add to dashboard Cite

Herdsmen’s access to credit funds plays an important supporting role in promoting the modern economic development of pastoral areas and broadening the channels for herders to increase their income. This paper studies the credit behavior of herdsmen from the perspective of emotional and psychological state identification to analyze the current credit behavior of herdsmen. Firstly, the internal mechanism of herdsmen’s credit and emotional and psychological identification research content are analyzed. Secondly, a scheme of emotional and psychological state identification is proposed, and a corresponding survey of herdsmen is carried out according to the scheme. Finally, the Probit and Heckman models are introduced to analyze the credit behavior of herdsmen. The results show that herdsmen with high emotional and psychological scores strongly demand credit. They are likely to have access to credit and have a high level of credit. Herdsmen with low scores of emotional and psychological characteristics have low or no apparent demand for credit, and their credit availability is also low. The herdsmen with high emotional and psychological characteristics have a 71% higher probability of generating credit demand than herdsmen with low openness. Their likelihood of obtaining credit is also 33.47% higher than herdsmen with low emotional and psychological characteristics. They also have easy access to higher loan amounts. The results provide a reference for related research and analysis of herdsman’s credit behavior and economic development.

show abstract

“…Recently, deep learning-based analysis of sentiments has achieved great success in different areas such as audiovisual [13], image [14], and sequential data [15] processing. While increasing the training data for traditional machine learning techniques does not always improve the performance, in deep learning, the success chance increases as the training data diversifies and increases.…”

Section: Natural Language Processing and Sentiment Analysismentioning

confidence: 99%

Deep learning-based user experience evaluation in distance learning

et al. 2023

View full text Add to dashboard Cite

The Covid-19 pandemic caused uncertainties in many different organizations, institutions gained experience in remote working and showed that high-quality distance education is a crucial component in higher education. The main concern in higher education is the impact of distance education on the quality of learning during such a pandemic. Although this type of education may be considered effective and beneficial at first glance, its effectiveness highly depends on a variety of factors such as the availability of online resources and individuals’ financial situations. In this study, the effectiveness of e-learning during the Covid-19 pandemic is evaluated using posted tweets, sentiment analysis, and topic modeling techniques. More than 160,000 tweets, addressing conditions related to the major change in the education system, were gathered from Twitter social network and deep learning-based sentiment analysis models and topic models based on latent dirichlet allocation (LDA) algorithm were developed and analyzed. Long short term memory-based sentiment analysis model using word2vec embedding was used to evaluate the opinions of Twitter users during distance education and also, a topic model using the LDA algorithm was built to identify the discussed topics in Twitter. The conducted experiments demonstrate the proposed model achieved an overall accuracy of 76%. Our findings also reveal that the Covid-19 pandemic has negative effects on individuals 54.5% of tweets were associated with negative emotions whereas this was relatively low on emotion reports in the YouGov survey and gender-rescaled emotion scores on Twitter. In parallel, we discuss the impact of the pandemic on education and how users’ emotions altered due to the catastrophic changes allied to the education system based on the proposed machine learning-based models.

show abstract

Deep learning based multimodal emotion recognition using model-level fusion of audio–visual modalities

Cited by 97 publications

References 31 publications

Subjective Evaluation of Basic Emotions from Audio–Visual Data

Subjective Evaluation of Basic Emotions from Audio–Visual Data

Analysis of Herdsmen’s Credit Behavior under the Identification Scheme of Emotional and Psychological State

Deep learning-based user experience evaluation in distance learning

Contact Info

Product

Resources

About