Emotion Understanding Using Multimodal Information Based on Autobiographical Memories for Alzheimer’s Patients

Abstract

Abstract. Alzheimer Disease (AD) early detection is considered of high importance for improving the quality of life of patients and their families. Amongst all the different approaches for AD detection, significant work has been focused on emotion analysis through facial expressions, body language or speech. Many studies also use the electroencephalogram in order to capture emotions that patients cannot physically express. Our work introduces an emotion recognition approach using facial expression and EEG signal analysis. A novel dataset was created specifically to remark the autobiographical memory deficits of AD patients. This work uses novel EEG features based on quaternions, facial landmarks and the combination of them. Their performance was evaluated in a comparative study with a state of the art methods that demonstrates the proposed approach.

A Survey of Alzheimer’s Disease Early Diagnosis Methods for Cognitive Assessment

“…Most research works focus on analysing whether AD patients are able to recognise facial expressions, but not many of them attempt to study the patients’ facial expressions in order to detect sudden mood changes or how they react to different stimuli. Fernández-Montenegro et al [ 54 ] propose a method that is based on the patients’ reactions to autobiographical stimuli. Using Spontaneous Multimodal Database (SEMdb) they propose novel EEG features that are based on quaternion PCA to classify reactions to recent and distant autobiographical memories and reactions to known and known people.…”

“…In [ 26 ] the authors used DTW, SVM and kNN binary classifiers to categorise gait data from AD and healthy subjects. The work in [ 54 ] classifies recent and long distance memories while using EEG and facial images using SVM and Boosting classifiers. In [ 51 ], the authors use Naïve Bayes classifier to distinguish healthy from MCI patients while they are reading text paragraphs using features, such as gaze duration, saccade amplitude, and the total number of fixations.…”

“…BRAINnet [ 79 ] is one of the largest Alzheimer’s databases containing general and cognitive information from interviews and cognitive testing applications, EEG, ERPs, autonomic arousal measures, MRI, and genomics. Spontaneous Multimodal Database (SEMdb) [ 54 ] is a multimodal dataset for spontaneous emotional reaction recognition that contains multimodal information (HD RGB, depth and IR frames of the face, EEG signal, and eye gaze data) of nine healthy participants: five females and four males between 27 to 60 years old with different educational backgrounds that were taken while completing cognitive/visual tests. The general purpose of this dataset is AD screening.…”

Characterizing the State of Apathy with Facial Expression and Motion Analysis

“…Though several biomarkers for apathy are discussed in Hampel et al [6], automated apathy diagnosis is a novel research area of high impact and hence interest. The computer vision based analysis of face and gesture has shown to provide abundant information about different neurodegenerative disorders [10], [11], [12], [13], which we here aim at exploiting for apathy diagnosis.…”

“…Facial expression recognition, as an indicator of internal emotional state, has been widely explored in the last two decades [17], [18], [19]. Montenegro et al [11] analyzed emotion recognition based on facial video and electroencephalograph signals for early detection of autobiographical memory deficits in AD. Similarly, mood disorders, such as major depressive disorder and bipolar disorder were investigated from facial expression analysis [20].…”