EEG-based detection of modality-specific visual and auditory sensory processing

Abstract

Objective. A passive brain-computer interface (pBCI) is a system that enhances a human-machine interaction by monitoring the mental state of the user and, based on this implicit information, making appropriate modifications to the interaction. Key to the development of such a system is the ability to reliably detect the mental state of interest via neural signals. Many different mental states have been investigated, including fatigue, attention and various emotions, however one of the most commonly studied states is mental workload, i.e., the amount of attentional resources required to perform a task. The emphasis of mental workload studies to date has been almost exclusively on detecting and predicting the "level" of cognitive resources required (e.g., high vs. low), but we argue that having information regarding the specific "type" of resources (e.g., visual or auditory) would allow the pBCI to apply more suitable adaption techniques than would be possible knowing just the overall workload level. Approach. 15 participants performed carefully designed visual and auditory tasks while EEG data was recorded. The tasks were designed to be as similar as possible to one another except for the type of attentional resources required. The tasks were performed at two different levels of demand. Using traditional machine learning algorithms, we investigated, firstly, if EEG can be used to distinguish between auditory and visual processing tasks and, secondly, what effect level of sensory processing demand has on the ability to distinguish between auditory and visual processing tasks. Main results. The results show that at the high level of demand, the auditory vs. visual processing tasks could be distinguished with an accuracy of 77.1% on average. However, in the low demand condition in this experiment, the tasks were not classified with an accuracy exceeding chance. Significance. These results support the feasibility of developing a pBCI for detecting not only the level, but also the type, of attentional resources being required of the user at a given time. Further research is required to determine if there is a threshold of demand under which the type of sensory processing cannot be detected, but even if that is the case, these results are still promising since it is the high end of demand that is of most concern in safety critical scenarios. Such a BCI could help improve safety in high risk occupations by initiating the most effective and efficient possible adaptation strategies when high workload conditions are detected.

EEG-based hierarchical classification of level of demand and modality of auditory and visual sensory processing

“…Three types of baseline trials were collected: one in which passive stimuli in both modalities were presented at the higher speed, one in which the passive stimuli in both modalities were presented at the lower speed, and a true baseline in which the participant focused on the black computer screen with a constant '+' symbol in the center. Because the results of our previous work suggested that our BL L condition was too similar to the Aud L and Vis L conditions [28], we excluded them (and the BL H condition) and only the true baseline (BL) trials were considered in this work.…”

“…high or low). This was because our previous results suggested that level of demand could be classified with better accuracy than type of demand [28], and it is good practice in hierarchical classification to put more accurate layers first to avoid error propagation. It consisted of a single classifier trained using data from all task conditions.…”

“…The fact that sensory processing was lower in low demand conditions was not unanticipated based on our previous work on this dataset [28]. In that work, we explored the ability to distinguish visual from auditory processing trials via EEG signals, in singlemodality conditions and within workload levels (i.e.…”

“…In a recent study [28], we sought to address some of these issues by investigating visual and auditory versions of a target monitoring task that differed only in the sensory modality of the target stimuli to be monitored. We also included passive/task-irrelevant stimuli from the opposing modality, so that we could be sure any ability to differentiate the tasks was indeed due to sensory processing/perception and not simply to the passive sensation of the stimuli.…”