Spectral EEG-based classification for operator dyads’ workload and cooperation level estimation

Abstract

There is a growing momentum to design online tools to measure mental workload for neuroergonomic purposes. Most of the research focuses on the monitoring of a single human operator. However, in real-life situations, human operators work in cooperation to optimize safety and performance. This is particularly the case in aviation whereby crews are composed of a pilot flying and a pilot monitoring. The motivation of this study is to evaluate the possibility to apply an hyperscanning approach to estimate the mental workload of crews composed of two operators. We designed an experimental protocol in which ten crews (i.e. 20 subjects) had to perform a modified version of the NASA MATB-II during 8 five-minute blocks (i.e. 4 mental workload level configurations * 2 cooperation v. non cooperation conditions). Mental workload and cooperation level were classified using a traditional passive brain-computer interface pipeline that includes a spatial filtering step on frequency features. Our results disclosed that all mental states' estimations were significantly above chance level. Intra-subject classification accuracy for mental workload (2 classes) was 63% for the pilot flying and 58% for the pilot monitoring. As for cooperation level, the binary classification reached 57% for the pilot flying and 60% for the pilot monitoring. Regarding the team, intra-team classification accuracy of the workload configuration of the team (4-class) reached 35%. As for the team cooperation level, the binary classifier reached 60% of accuracy. The results are discussed in terms of hyperscanning applications.

Team Workload in Action Teams: Exploring the Impact of Interdependence

“…Demand manipulations in previous action team experiments have spanned across dimensions including task quantity (Strang et al, 2012), task difficulty (Verdière et al, 2019), time pressure (Cui et al, 2021), demand imbalances (Porter et al, 2010), workload transitions and non-routine events (Jobidon et al, 2006), and asset availability (Knott et al, 2006). For this study, we wanted to manipulate demand (1) at the team-level and (2) in a resource-limited manner (also at the team level).…”

Automatic Estimation of Interpersonal Engagement During Naturalistic Conversation Using Dyadic Physiological Measurements

“…However, in the future, we will also explore an alternative study protocol where dyads will go through multiple artificially induced conversation scenarios (e.g., told to argue with each other), and classification methods will be used to assign physiological data to one of the possible scenarios. While less natural than the current protocol, this is likely to provide more balanced data, and classification algorithms are more common than regression algorithms in both studies of physiological synchrony (Hernandez et al, 2014;Konvalinka et al, 2014;Muszynski et al, 2018;Zhu et al, 2018;Brouwer et al, 2019;Verdiere et al, 2019;Darzi and Novak, 2021) and general affective computing (Novak et al, 2012;Aranha et al, 2019). Unrelated to the above classification approach, we may also consider a multi-day protocol where engagement estimation algorithms are trained on data from one session, then tested on data from another session.…”

“…However, while there have been many studies targeting group-level analysis of physiological synchrony (e.g., correlating synchrony and engagement in a large sample), there has been relatively little work on quantifying engagement or other interpersonal states at the level of individual dyads (e.g., tracking interpersonal engagement of a specific dyad over time). A handful of studies have used classification algorithms with a single physiological modality (e.g., electroencephalography alone) to discriminate between two states (e.g., engaged vs. unengaged dyads) ( Hernandez et al, 2014 ; Konvalinka et al, 2014 ; Muszynski et al, 2018 ; Zhu et al, 2018 ; Brouwer et al, 2019 ; Pan et al, 2020 ) with one study discriminating between four affective states ( Verdiere et al, 2019 ). A final study used regression algorithms to map physiological synchrony to self-reported arousal and valence on 1–9 scales using electroencephalography during video watching ( Ding et al, 2021 ).…”

Near-Zero Phase-Lag Hyperscanning in a Novel Wireless EEG System

“…One study [21] used LabRecorder [20], the main recording application of LSL, to synchronize and centralize the streams of two wireless EEG systems on a network while research participants performed a word-by-word interaction task. LSL-synchronized EEG systems were used in another study [22] to explore the mental workload of one pilot flying and another pilot monitoring during a simulated flight. With the support of cross-platform development and the need for multimodal data streaming, an increasing number of new devices feature an LSL plugin to enable two-way communication between devices with submillisecond timing precision.…”