Assessing the Effects of Various Physiological Signal Modalities on Predicting Different Human Cognitive States

Abstract

Robust estimation of systemic human cognitive states is critical for a variety of applications, from simply detecting inefficiencies in task assignments, to the adaptation of artificial agents’ behaviors to improve team performance in mixed-initiative human-machine teams. This study showed that human eye gaze, in particular, the percentage change in pupil size (PCPS), is the most reliable biomarker for assessing three human cognitive states including workload, sense of urgency, and mind wandering compared to electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS), respiration, and skin conductance. We used comprehensive multi-modal driving dataset to examine the accuracy of signals to assess these cognitive states. We performed comprehensive statistical tests to validate the performance of several physiological signals to determine human cognitive states and demonstrated that PCPS shows noticeably superior performance. We also characterized the link between workload and sense of urgency with eye gaze and observed that consecutive occurrences of higher sense of urgency were prone to increase overall workload. Finally, we trained five machine learning (ML) models and showed that four of them had similar accuracy in cognitive state classification (with one, random forest, showing inferior performance). The results provided evidence that the PCPS is a reliable physiological marker for cognitive state estimation.