Method for Improving EEG Based Emotion Recognition by Combining It with Synchronized Biometric and Eye Tracking Technologies in a Non-invasive and Low Cost Way

Abstract

Technical advances, particularly the integration of wearable and embedded sensors, facilitate tracking of physiological responses in a less intrusive way. Currently, there are many devices that allow gathering biometric measurements from human beings, such as EEG Headsets or Health Bracelets. The massive data sets generated by tracking of EEG and physiology may be used, among other things, to infer knowledge about human moods and emotions. Apart from direct biometric signal measurement, eye tracking systems are nowadays capable of determining the point of gaze of the users when interacting in ICT environments, which provides an added value research on many different areas, such as psychology or marketing. We present a process in which devices for eye tracking, biometric, and EEG signal measurements are synchronously used for studying both basic and complex emotions. We selected the least intrusive devices for different signal data collection given the study requirements and cost constraints, so users would behave in the most natural way possible. On the one hand, we have been able to determine basic emotions participants were experiencing by means of valence and arousal. On the other hand, a complex emotion such as empathy has also been detected. To validate the usefulness of this approach, a study involving forty-four people has been carried out, where they were exposed to a series of affective stimuli while their EEG activity, biometric signals, and eye position were synchronously recorded to detect self-regulation. The hypothesis of the work was that people who self-regulated would show significantly different results when analyzing their EEG data. Participants were divided into two groups depending on whether Electro Dermal Activity (EDA) data indicated they self-regulated or not. The comparison of the results obtained using different machine learning algorithms for emotion recognition shows that using EEG activity alone as a predictor for self-regulation does not allow properly determining whether a person in self-regulation its emotions while watching affective stimuli. However, adequately combining different data sources in a synchronous way to detect emotions makes it possible to overcome the limitations of single detection methods.

Review and Classification of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review

“…For example, combining eye movements, which are measured using an eye tracking method, and EEG can considerably improve the performance of emotion recognition systems [38,39]. Moreover, in [40], the researchers proposed a multi-modal emotion recognition system using four sensing methods: EEG, heart inter-beat interval, galvanic skin response, and stressor level lever.…”

“…These proposed systems aim to explore or improve EEG-based emotion recognition systems. [2,39,41,42,49,50,57,61,63,92,104,108,109,117,131,136,149,152,157,173,174,185,186,189,191,[195][196][197][198][199][200][201][202][203][204][205][206][207][208][209]217,219,[223][224][225]229,[262][263][264][265][266]<...>…”

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

“…Therefore, eye movement data, as a behavioral reaction to emotions, have been widely utilized to assist with EEG-based emotion recognition in aBCI systems. López-Gil et al [18] improved EEG-based emotion recognition by combining eye tracking and synchronized biometrics to detect the valence-arousal basic emotions and a complex emotion of empathy. Zheng et al [50] evaluated the complementary characteristics of EEG and eye movement data in classifying positive, neutral and negative emotions by fusing the DE and pupil diameter features.…”

“…underlying emotion [15] [16]. Moreover, the fusion of EEG and eye tracking data has been shown efficient in multimodal emotion recognition, with increasing interests among research communities [17] [18]. In this paper, we adopt the EEG signals, along with eye movement data or peripheral physiological signals, to classify different emotions.…”

“…Most existing studies on EEG-based emotion recognition have relied on single-channel analysis [15], [16], [17], [18], where EEG features are independently extracted within each EEG channel in different brain regions. In contrast, studies on cognitive science and neuroimaging have demonstrated that emotion is a complex behavioral and physiological reaction that involves circuits in multiple cerebral regions [19].…”