Unsupervised adaptation of an ECoG based brain–computer interface using neural correlates of task performance

Abstract

Brain–computer interfaces (BCIs) translate brain signals into commands to external effectors, and mainly target severely disabled users. The usability of BCIs may be improved by reducing their major constraints, such as the necessity for special training sessions to initially calibrate and later keep up to date the neural signal decoders. In this study, we show that it is possible to train and update BCI decoders during free use of motor BCIs. In addition to the neural signal decoder controlling effectors (control decoder), one more classifier is proposed to detect neural correlates of BCI motor task performances (MTP). MTP decoders reveal whether the actions performed by BCI effectors matched the user’s intentions. The combined outputs of MTP and control decoders allow forming training datasets to update the control decoder online and in real time during free use of BCIs. The usability of the proposed auto-adaptive BCI (aaBCI) is demonstrated for two principle BCIs paradigms: with discrete outputs (4 classes BCI, virtual 4-limb exoskeleton), and with continuous outputs (cursor 2D control). The proof of concept was performed in an online simulation study using an ECoG dataset collected from a tetraplegic during a BCI clinical trial. The control decoder reached a multiclass area under the ROC curve of 0.7404 using aaBCI, compared to a chance level of 0.5173 and to 0.8187 for supervised training for the multiclass BCI, and a cosine similarity of 0.1211 using aaBCI, compared to a chance level of 0.0036 and to 0.2002 for supervised training for the continuous BCI.

Online adaptive group-wise sparse Penalized Recursive Exponentially Weighted N-way Partial Least Square for epidural intracranial BCI

“…It was applied in real time for closed-loop adaptation of 3D hand translation/wrist rotation decoders in tetraplegics (Benabid et al, 2019 ; Moly et al, 2022 ). Finally, REW-NPLS was tested in the simulation of auto-adaptive continuous (bi-directional cursor control) and discrete multiclass motor imagery (MI) BCI in a tetraplegic patient (Rouanne et al, 2022 ). In the (Sliwowski et al, 2022 ) offline study, ANN algorithms were reported to outperform the REW-NPLS decoder.…”

A spiking neural network with continuous local learning for robust online brain machine interface

“…While offline experiments on pre-recorded data are commonly used to evaluate neural decoder performance, they do not always correlate well with performance in an online, closed-loop setting [24][25][26]. In a closed loop environment, the output of the BMI is fed directly to the output device, allowing the user to attempt to correct for any errors in the BMI output in real time.…”

“…In the first experiment, 30 out of 46 input neurons were removed after 50 reaches. This simulates a loss of connection to neurons that is common in real BMI systems due to electrodes shifting or becoming damaged [24,49]. The continuous learning SNN was allowed to train using continuous learning for 30 subsequent reaches, while the other decoders were recalibrated on neural data from the first 30 reaches after reassignment.…”

“…In the second experiment, the preferred directions of 30 neurons were randomly reassigned. Although real neurons do not suddenly change behavior like this, it is common for electrodes used in implantable BMI to shift during use and start picking up data from a new set of neurons [24,27], which can be simulated by changing the properties of the existing neurons. All decoders showed a large decrease in quality following neural reassignment.…”

“…Additionally, SNN decoders have not been tested in an online BMI setting, in which the user attempts to use the BMI in real time and receives information about the output of the BMI. Neural decoders trained offline can have quite different results when applied an online setting due to the adjustments made by the user in response to feedback from the BMI system [24][25][26]. Even decoders that perform well initially in an online setting may see a decrease in performance over time due to changes in the structure of nearby neurons or sudden loss of neurons [27,28].…”