Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

Zhang, Xiaowei; Liu, Jinyong; Shen, Jian; Li, Shaojie; Hou, Kechen; Hu, Bin; Gao, Jin; Zhang, Tong

doi:10.1109/tcyb.2020.2987575

Cited by 104 publications

(55 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multimodal ML is a research field with one of its earliest application in the field of Audio-Visual Speech Recognition (AVSR) [8], to contemporary fields of emotion recognition [9], [10] and tracking multiple agents [11] or Anomalies in traffic [12]. Like sensor data fusion, in the sense that it uses multiple sensor modalities, Multimodal ML uses multiple ML algorithms, either in parallel or in series, to improve a framework [13].…”

Section: Related Workmentioning

confidence: 99%

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

Roche

De-Silva

Hook

et al. 2022

IEEE Trans. Cybern.

View full text Add to dashboard Cite

Increasingly, the task of detecting and recognising the actions of a human has been delegated to some form of neural network processing camera or wearable sensor data. Due to the degree to which the camera can be affected by lighting and wearable sensors scantiness, neither one modality can capture the required data to perform the task confidently. That being the case, range sensors like Light Detection and Ranging (LiDAR) can complement the process to perceive the environment more robustly. Most recently, researchers have been exploring ways to apply Convolutional Neural Networks to 3-Dimensional data. These methods typically rely on a single modality and cannot draw on information from complementing sensor streams to improve accuracy. This paper proposes a framework to tackle human activity recognition by leveraging the benefits of sensor fusion and multimodal machine learning. Given both RGB and point cloud data, our method describes the activities being performed by subjects using regions with a convolutional neural network (R-CNN) and a 3-Dimensional modified Fisher vector network. Evaluated on a custom captured multimodal dataset demonstrates that the model outputs remarkably accurate human activity classification (90%). Furthermore, this framework can be used for sports analytics, understanding social behaviour, surveillance, and perhaps most notably by Autonomous Vehicles (AV) to datadriven decision-making policies in urban areas and indoor environments.

show abstract

Section: Related Workmentioning

confidence: 99%

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

Roche

De-Silva

Hook

et al. 2022

IEEE Trans. Cybern.

View full text Add to dashboard Cite

show abstract

“…The neural network is implemented based on linear weighted decisionmaking stage fusion, which increases efficiency and precision. [69] proposed a revolutionary multimodal fusion system with regularization focused on a new kernel matrix perspective and a deep network architecture. In representation learning, they used the deep network architecture's superior efficiency to transform the native space of a predefined kernel into a task-specific function space.…”

Section: A Multimodal Emotion Recognition Combining (Audiomentioning

confidence: 99%

Multimodal Emotion Recognition using Deep Learning

Abdullah

Ameen

Sadeeq

et al. 2021

JASTT

168

View full text Add to dashboard Cite

New research into human-computer interaction seeks to consider the consumer's emotional status to provide a seamless human-computer interface. This would make it possible for people to survive and be used in widespread fields, including education and medicine. Multiple techniques can be defined through human feelings, including expressions, facial images, physiological signs, and neuroimaging strategies. This paper presents a review of emotional recognition of multimodal signals using deep learning and comparing their applications based on current studies. Multimodal affective computing systems are studied alongside unimodal solutions as they offer higher accuracy of classification. Accuracy varies according to the number of emotions observed, features extracted, classification system and database consistency. Numerous theories on the methodology of emotional detection and recent emotional science address the following topics. This would encourage studies to understand better physiological signals of the current state of the science and its emotional awareness problems.

show abstract

“…After a long period of human evolution, the visual system has evolved a variety of neural mechanisms to adapt to the surrounding environment, each of which can efficiently process specific visual information. These specific visual information include position, form, texture, and movement ( Kim et al, 2016 ; Zhang X. et al, 2020 ). In addition, the processing of visual information has the characteristic of filtering, which is accompanied by the immediate focus of human attention.…”

Section: Research Foundationmentioning

confidence: 99%

Visual Expression of Emotion in Dynamic 3D Painting System Based on Emotion Synthesis Model

Cheng

2021

Front. Psychol.

View full text Add to dashboard Cite

Emotion is a unique ability possessed by human beings as advanced creatures. Emotions give people a unique physical and mental experience. Assigning emotions to computer systems is one of the latest topics in artificial intelligence research. The purpose is to allow machines to achieve natural coordination between humans and computers. This article focuses on the visual expression of emotion in the dynamic three-dimensional painting system, creating an intelligent painting system and realizing a good user experience. In this paper, the discrete method is used to qualitatively analyze emotions, and the continuous method is used to quantify basic emotions, and emotional modeling and emotional quantitative analysis are proposed to realize quantitative analysis of emotions. Combining these two methods, a comprehensive method is proposed, which uses a continuous method to quantify the basic emotions of each discrete dimension, and finally superimposes them into a comprehensive emotional synthesis model. Emotion modeling is the basis of emotion visualization. Borrowing the relationship between emotion synthesis model and visual emotion elements, this article puts forward the concept of qualitative and quantitative visual emotion elements, and expounds that the multidimensional superposition of visual emotion elements makes dynamic three-dimensional painting system emotions. The experimental results in this article show that the emotional visualization scheme of 100 samples is tested by quantitative statistical methods to demonstrate its effectiveness. Starting from 5 points of concern, the emotion visualization method discussed in this article can indeed convey or suggest a certain positive emotion (the average value of experience, transitivity, and infectiousness > 2.5, and the variance is close to 0), but we also found this recognition at the same time The degree is not high enough, and individual differences are large (mean value < 2.5, variance close to 1). This can indicate that different subjects have different feelings and evaluations of this emotional visualization. As long as the difference is within a reasonable range, this emotional visualization also has practical value, and has the ability to convey or suggest emotions.

show abstract

Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

Cited by 104 publications

References 34 publications

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

Multimodal Emotion Recognition using Deep Learning

Visual Expression of Emotion in Dynamic 3D Painting System Based on Emotion Synthesis Model

Contact Info

Product

Resources

About