Automated analysis of facial emotions in subjects with cognitive impairment

Abstract

Differences in expressing facial emotions are broadly observed in people with cognitive impairment. However, these differences have been difficult to objectively quantify and systematically evaluate among people with cognitive impairment across disease etiologies and severity. Therefore, a computer vision-based deep learning model for facial emotion recognition trained on 400.000 faces was utilized to analyze facial emotions expressed during a passive viewing memory test. In addition, this study was conducted on a large number of individuals (n = 493), including healthy controls and individuals with cognitive impairment due to diverse underlying etiologies and across different disease stages. Diagnoses included subjective cognitive impairment, Mild Cognitive Impairment (MCI) due to AD, MCI due to other etiologies, dementia due to Alzheimer’s diseases (AD), and dementia due to other etiologies (e.g., Vascular Dementia, Frontotemporal Dementia, Lewy Body Dementia, etc.). The Montreal Cognitive Assessment (MoCA) was used to evaluate cognitive performance across all participants. A participant with a score of less than or equal to 24 was considered cognitively impaired (CI). Compared to cognitively unimpaired (CU) participants, CI participants expressed significantly less positive emotions, more negative emotions, and higher facial expressiveness during the test. In addition, classification analysis revealed that facial emotions expressed during the test allowed effective differentiation of CI from CU participants, largely independent of sex, race, age, education level, mood, and eye movements (derived from an eye-tracking-based digital biomarker for cognitive impairment). No screening methods reliably differentiated the underlying etiology of the cognitive impairment. The findings provide quantitative and comprehensive evidence that the expression of facial emotions is significantly different in people with cognitive impairment, and suggests this may be a useful tool for passive screening of cognitive impairment.

Beyond shallow feelings of complex affect: Non-motor correlates of subjective emotional experience in Parkinson’s disease

“…Moreover, in combination with delayed verbal recall and semantic fluency, the ability to experience tender feelings explained 83% of the variance in global cognition among patients without prominent frontal damage. We found Tenderness experience and recall memory to have a similar though independent contribution to global cognition, in line with prior work highlighting that disturbances in memory and emotion are independently related to cognitive impairment [125]. The interrelation we report here may reflect the discrete contribution of distinct functional networks, which subserve affiliative behavior, memory, and semantic cognition, to overall BF neuromodulatory dynamics [59,72,126,127].…”

An investigation of privacy preservation in deep learning-based eye-tracking

“…The data set used in this work contains 610 video recordings from 493 participants in the Emory Healthy Aging Study undergoing an eye-tracking-based evaluation of neurological function and are described in Haque et al [ 16 , 21 ] and Jiang et al [ 20 ]. The videos are recorded in 30 frames per second and are closeups of participants.…”

“…Convolutional neural networks (CNN) can be particularly complex. The increased adoption of CNNs in the context of facial analysis and medical imaging [ 19 , 20 ] raises concerns over their ability to encode private data. This work, therefore, explores a CNN-based model to stress-test under inference attacks, developed for an eye-tracking task [ 21 ], designed to estimate the severity of illness in cognitively impaired individuals [ 21 ].…”

Multimodal Assessment of Schizophrenia and Depression Utilizing Video, Acoustic, Locomotor, Electroencephalographic, and Heart Rate Technology: Protocol for an Observational Study

“…The cost of powerful computing hardware has fallen, and improvements in the field of computer science and health care suggest that various types of computer sensors and recording hardware could be used to aid in the assessment and diagnostic prediction of mental illness [25][26][27][28][29][30]. Research groups have demonstrated the efficacy in numerous mental health populations of these technologies, which include computer vision for distinguishing phases of depression [25,26], schizophrenia [27], and cognitive impairment [31], actigraphy for the differentiation of patients with schizophrenia from controls [28,29], and heart rate monitoring for distinguishing patients with schizophrenia or posttraumatic stress disorder from controls [28][29][30]. This research has demonstrated that with heart rate variability and actigraphic assessments alone, patients with schizophrenia may be differentiated from controls with up to 95.3% accuracy [28,29].…”