In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI

Abstract

There is growing interest for implementing tools to monitor cognitive performance in naturalistic work and everyday life settings. The emerging field of research, known as neuroergonomics, promotes the use of wearable and portable brain monitoring sensors such as functional near infrared spectroscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. The objective of this study was to implement an on-line passive fNIRS-based brain computer interface to discriminate two levels of working memory load during highly ecological aircraft piloting tasks. Twenty eight recruited pilots were equally split into two groups (flight simulator vs. real aircraft). In both cases, identical approaches and experimental stimuli were used (serial memorization task, consisting in repeating series of pre-recorded air traffic control instructions, easy vs. hard). The results show pilots in the real flight condition committed more errors and had higher anterior prefrontal cortex activation than pilots in the simulator, when completing cognitively demanding tasks. Nevertheless, evaluation of single trial working memory load classification showed high accuracy (>76%) across both experimental conditions. The contributions here are two-fold. First, we demonstrate the feasibility of passively monitoring cognitive load in a realistic and complex situation (live piloting of an aircraft). In addition, the differences in performance and brain activity between the two experimental conditions underscore the need for ecologically-valid investigations.

A Neuroergonomics Approach to Mental Workload, Engagement and Human Performance

“…The PFC serves a control function during routine cognitive operations, such as: action selection, retrieval/updating in working memory, monitoring and inhibition (Ramnani and Owen, 2004;Ridderinkhof et al, 2004). It is often activated during high levels of cognitive demand (Ayaz et al, 2012;Herff et al, 2014;Racz et al, 2017;Gateau et al, 2018;Fairclough et al, 2019) and dysfunction of this structure is known to degrade performance (Sandson and Albert, 1984;Dolcos and McCarthy, 2006). However, the PFC is complex and its function is subject to the quadratic influence of neuromodulation via the influence of noradrenaline and dopamine (Arnsten, 2009;Arnsten et al, 2012).…”

“…The implementation of such neuro-adaptive technology relies on a pipeline that consists of a signal acquisition step, a preprocessing step to improve the signal-to-noise ratio, a feature extraction step, a classification step to diagnose the current mental states, and lastly an adaptation step (Zander and Kothe, 2011;Roy and Frey, 2016). This last step implies the implementation of formal decisional unit (Gateau et al, 2018) that dynamically close the loop by triggering the most appropriate cognitive countermeasures (May and Baldwin, 2009). There are currently three types of mitigating solutions to instigate a change in behaviors via: (1) adaptation of the user interface, (2) adaptation of the task and of the level automation, and the (3) "neuroadaptation" of the end-users.…”

“…Transcranial direct current stimulation (tDCS) represents a technique of neuromodulation that can be used to boost executive functioning (see Callan and Perrey, 2019;Cinel et al, 2019). This portable device can be combined with EEG and fNIRS and used in the context of real-life task performance for the purpose of on-line neuromodulation (McKendrick et al, 2015;Gateau et al, 2018). For example, a number of studies support the position that neurostimulation can: enhance mental flexibility and mitigate perseveration (Leite et al, 2011;Jeon and Han, 2012), improve visual attention (Falcone et al, 2012;Nelson et al, 2015), improve executive functioning in multitasking situations (Nelson et al, 2016) and increase alertness (McIntire et al, 2014;Nelson et al, 2014).…”

Application of functional near-infrared spectroscopy in the healthcare industry: A review

“…365 The acquisition of data from a realistic environment and its processing are vital for the development of BCI. 366,367 Research on the mental states and neuroergonomics of pilots during actual as well as simulated environment was conducted using fNIRS. 368,369 Satisfactory results from these studies strengthened the use of fNIRS in BCI applications.…”

Impact of Tea and Coffee Consumption on Cognitive Performance: An fNIRS and EDA Study

“…This method uses a holistic approach, combining different metrics from behavioral performance, physiological measures, and measures from the brain directly to give a more comprehensive overview beyond that of traditional methods [26,28]. The development of miniaturized, portable, wireless, and wearable sensors has allowed for continuous monitoring of participants while they perform real-world tasks in everyday settings [28][29][30][31][32].…”