Reinforcement learning for adaptive threshold control of restorative brain-computer interfaces: a Bayesian simulation

Abstract

Restorative brain-computer interfaces (BCI) are increasingly used to provide feedback of neuronal states in a bid to normalize pathological brain activity and achieve behavioral gains. However, patients and healthy subjects alike often show a large variability, or even inability, of brain self-regulation for BCI control, known as BCI illiteracy. Although current co-adaptive algorithms are powerful for assistive BCIs, their inherent class switching clashes with the operant conditioning goal of restorative BCIs. Moreover, due to the treatment rationale, the classifier of restorative BCIs usually has a constrained feature space, thus limiting the possibility of classifier adaptation. In this context, we applied a Bayesian model of neurofeedback and reinforcement learning for different threshold selection strategies to study the impact of threshold adaptation of a linear classifier on optimizing restorative BCIs. For each feedback iteration, we first determined the thresholds that result in minimal action entropy and maximal instructional efficiency. We then used the resulting vector for the simulation of continuous threshold adaptation. We could thus show that threshold adaptation can improve reinforcement learning, particularly in cases of BCI illiteracy. Finally, on the basis of information-theory, we provided an explanation for the achieved benefits of adaptive threshold setting.

What is the optimal task difficulty for reinforcement learning of brain self-regulation?

“…difficulty levels. We have, therefore, suggested to interpret the classification accuracy curve over thresholds (see figure 1), where a classification accuracy above 50% is achieved, as indicative of the ZPD (Bauer and Gharabaghi 2015a), a perspective which is in line with the Bayesian reinforcement learning model (Bauer and Gharabaghi 2015b). …”

“…From a learning perspective, however, the classification accuracy indicates whether the reward is specific and sensitive. We explored, therefore, a mathematical simulation based on a Bayesian reinforcement learning model to find out which thresholds are optimal for learning (Bauer and Gharabaghi 2015b). This model revealed that learning occurred earliest at the threshold of maximum classification accuracy (θ CA ).…”

“…This model revealed that learning occurred earliest at the threshold of maximum classification accuracy (θ CA ). The mathematical modeling indicated, moreover, that operant conditioning can be optimized when an adaptation strategy for threshold selection is applied in the course of the training (Bauer and Gharabaghi, 2015b). Such an adaptation strategy would need to change the classifier threshold, i.e., difficulty level, of the feedback device to challenge the participant in the course of the training.…”

Therapeutic applications of BCI technologies

“…By providing feedback of the user’s own brain activity in real-time that are associated with specific neural states, BCIs reinforce the targeted brain states or dynamic modulations for achieving behavioral gains [75,77-78]. Despite the recent surge in BCI research for neurobehavioral modulation and previous reports of reduced disease symptoms, EEG neurofeedback has not been widely adopted as a therapeutic modality.…”

Closed-Loop Task Difficulty Adaptation during Virtual Reality Reach-to-Grasp Training Assisted with an Exoskeleton for Stroke Rehabilitation

“…This training approach was potentially too challenging and may even have frustrated the patients (Fels et al, 2015). The patients' cognitive resources for coping with the mental load of performing such a neurofeedback task must therefore be taken into consideration (Bauer and Gharabaghi, 2015a; Naros and Gharabaghi, 2015). Mathematical modeling on the basis of Bayesian simulation indicates that this might be achieved when the task difficulty is adapted in the course of the training (Bauer and Gharabaghi, 2015b).…”