A systematic review on hybrid EEG/fNIRS in brain-computer interface

Liu, Ziming; Shore, Jeremy; Wang, Miao; Yuan, Fengpei; Buss, Aaron T.; Zhao, Xiaopeng

doi:10.1016/j.bspc.2021.102595

Cited by 59 publications

(42 citation statements)

References 109 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Among them, EEG and fNIRS are the foremost modalities in terms of cost and manageability (Rahman et al, 2020 ; Rashid et al, 2020 ). EEG measures brain activity by calculating the voltage fluctuations from the action potentials of neurons, whereas fNIRS detects brain activity related to hemodynamic response changes (Hong and Zafar, 2018 ; Liu et al, 2021 ). Although invasive techniques provide more accurate data than non-invasive techniques, non-invasive modalities are more frequent and appreciated in the research domain.…”

Section: Introductionmentioning

confidence: 99%

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

et al. 2022

View full text Add to dashboard Cite

The constantly evolving human–machine interaction and advancement in sociotechnical systems have made it essential to analyze vital human factors such as mental workload, vigilance, fatigue, and stress by monitoring brain states for optimum performance and human safety. Similarly, brain signals have become paramount for rehabilitation and assistive purposes in fields such as brain–computer interface (BCI) and closed-loop neuromodulation for neurological disorders and motor disabilities. The complexity, non-stationary nature, and low signal-to-noise ratio of brain signals pose significant challenges for researchers to design robust and reliable BCI systems to accurately detect meaningful changes in brain states outside the laboratory environment. Different neuroimaging modalities are used in hybrid settings to enhance accuracy, increase control commands, and decrease the time required for brain activity detection. Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) measure the hemodynamic and electrical activity of the brain with a good spatial and temporal resolution, respectively. However, in hybrid settings, where both modalities enhance the output performance of BCI, their data compatibility due to the huge discrepancy between their sampling rate and the number of channels remains a challenge for real-time BCI applications. Traditional methods, such as downsampling and channel selection, result in important information loss while making both modalities compatible. In this study, we present a novel recurrence plot (RP)-based time-distributed convolutional neural network and long short-term memory (CNN-LSTM) algorithm for the integrated classification of fNIRS EEG for hybrid BCI applications. The acquired brain signals are first projected into a non-linear dimension with RPs and fed into the CNN to extract essential features without performing any downsampling. Then, LSTM is used to learn the chronological features and time-dependence relation to detect brain activity. The average accuracies achieved with the proposed model were 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. Moreover, the maximum accuracies achieved were 85.9, 88.1, and 92.4%, respectively. The results confirm the viability of the RP-based deep-learning algorithm for successful BCI systems.

show abstract

Section: Introductionmentioning

confidence: 99%

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The present study also contributes to further promoting the use of rmPFC signals in BCI applications. It has been demonstrated that the PFC regions contain sufficient information to accurately detect brain processes, including sensorimotor processes, cognitive functions, and mental states (Min et al, 2017 ; Khan et al, 2021 ; Liu et al, 2021 ). Our study has untangled one of the rmPFC's roles, which is related to stimulus-directed cognitive states.…”

Section: Discussionmentioning

confidence: 99%

Involvement of the Rostromedial Prefrontal Cortex in Human-Robot Interaction: fNIRS Evidence From a Robot-Assisted Motor Task

Watanabe

Ogawa

et al. 2022

Front. Neurorobot.

View full text Add to dashboard Cite

Assistive exoskeleton robots are being widely applied in neurorehabilitation to improve upper-limb motor and somatosensory functions. During robot-assisted exercises, the central nervous system appears to highly attend to external information-processing (IP) to efficiently interact with robotic assistance. However, the neural mechanisms underlying this process remain unclear. The rostromedial prefrontal cortex (rmPFC) may be the core of the executive resource allocation that generates biases in the allocation of processing resources toward an external IP according to current behavioral demands. Here, we used functional near-infrared spectroscopy to investigate the cortical activation associated with executive resource allocation during a robot-assisted motor task. During data acquisition, participants performed a right-arm motor task using elbow flexion-extension movements in three different loading conditions: robotic assistive loading (ROB), resistive loading (RES), and non-loading (NON). Participants were asked to strive for kinematic consistency in their movements. A one-way repeated measures analysis of variance and general linear model-based methods were employed to examine task-related activity. We demonstrated that hemodynamic responses in the ventral and dorsal rmPFC were higher during ROB than during NON. Moreover, greater hemodynamic responses in the ventral rmPFC were observed during ROB than during RES. Increased activation in ventral and dorsal rmPFC subregions may be involved in the executive resource allocation that prioritizes external IP during human-robot interactions. In conclusion, these findings provide novel insights regarding the involvement of executive control during a robot-assisted motor task.

show abstract

“…Processing neural signals requires machine learning techniques. According to Liu et al machine learning is widely used in hybrid EEG-NIRS systems that detect muscle intentions to produce categorizations that assist in providing motion control, emotional states, and visual attention [91]. As the feasibility of meaningful signal increases, the desire to provide team awareness of signals may also grow.…”

Section: Aismentioning

confidence: 99%

Characterizing Work-Life for Information Work on Mars

Linder

Hunter

McLemore

et al. 2022

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

We present a design fiction, which is set in the near future as significant Mars habitation begins. Our goal in creating this fiction is to address current work-life issues on Earth and Mars in the future. With shelter-in-place measures, established norms of productivity and relaxation have been shaken. The fiction creates an opportunity to explore boundaries between work and life, which are changing with shelter-in-place and will continue to change. Our work includes two primary artifacts: (1) a propaganda recruitment poster and (2) a fictional narrative account. The former paints the work-life on Mars as heroic, fulfilling, and fun. The latter provides a contrast that depicts the lived experience of early Mars inhabitants. Our statement draws from our design fiction in order to reflect on the structure of work, stress identification and management, family and work-family communication, and the role of automation.

show abstract

A systematic review on hybrid EEG/fNIRS in brain-computer interface

Cited by 59 publications

References 109 publications

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

Involvement of the Rostromedial Prefrontal Cortex in Human-Robot Interaction: fNIRS Evidence From a Robot-Assisted Motor Task

Characterizing Work-Life for Information Work on Mars

Contact Info

Product

Resources

About