Control of a center-out reaching task using a reinforcement learning Brain-Machine Interface

Sanchez, Justin C.; Tarigoppula, Aditya; Choi, J. S.; Marsh, B. T.; Chhatbar, Pratik Y.; Mahmoudi, Babak; Francis, Joseph T.

doi:10.1109/ner.2011.5910601

Cited by 34 publications

(17 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The assignment of reward is based on the 1-0 distance to the target, that is, dist⁡( x , d ) = 0 if x = d , and dist⁡( x , d ) = 1, otherwise. Once the cursor reaches the assigned target, the agent gets a positive reward +0.6, else it receives negative reward −0.6 [35]. Exploration rate ϵ = 0.01 and discount factor γ = 0.9 are applied.…”

Section: Experimental Results On Neural Decodingmentioning

confidence: 99%

Kernel Temporal Differences for Neural Decoding

Bae

Giraldo

Pohlmeyer

et al. 2015

Computational Intelligence and Neuroscience

Self Cite

View full text Add to dashboard Cite

We study the feasibility and capability of the kernel temporal difference (KTD)(λ) algorithm for neural decoding. KTD(λ) is an online, kernel-based learning algorithm, which has been introduced to estimate value functions in reinforcement learning. This algorithm combines kernel-based representations with the temporal difference approach to learning. One of our key observations is that by using strictly positive definite kernels, algorithm's convergence can be guaranteed for policy evaluation. The algorithm's nonlinear functional approximation capabilities are shown in both simulations of policy evaluation and neural decoding problems (policy improvement). KTD can handle high-dimensional neural states containing spatial-temporal information at a reasonable computational complexity allowing real-time applications. When the algorithm seeks a proper mapping between a monkey's neural states and desired positions of a computer cursor or a robot arm, in both open-loop and closed-loop experiments, it can effectively learn the neural state to action mapping. Finally, a visualization of the coadaptation process between the decoder and the subject shows the algorithm's capabilities in reinforcement learning brain machine interfaces.

show abstract

Section: Experimental Results On Neural Decodingmentioning

confidence: 99%

Kernel Temporal Differences for Neural Decoding

Bae

Giraldo

Pohlmeyer

et al. 2015

Computational Intelligence and Neuroscience

Self Cite

View full text Add to dashboard Cite

show abstract

“…RL has also been applied in BMI research. Concentrating mainly on controlling (prosthetic/robotic) devices, several studies have been reported, including: mapping neural activity to intended behavior through coadaptive BMI (using TD(λ)) [190] and symbiotic BMI (using actor-critic) [191], a testbed targeting center-out reaching task in primates for creating more realistic BMI control models [192], Hebbian RL for adaptive control by mapping neural states to prosthetic actions [193], BMI for unsupervised decoding of cortical spikes in multistep goal-directed tracking task (using Q(λ)) [194], adaptive BMI capable of adjusting to dramatic reorganizing neural activities with minimal training and stable performance over long duration (using actor-critic) [195], 11/33 BMI for efficient nonlinear mapping of neural states to actions through sparsification of state-action mapping space using quantized attention-gated kernel RL as an approximator [196]. Also, Lampe et al proposed BMI capable of transmitting imaginary movements evoked EEG signals over the Internet to remotely control robotic device [182], and Bauer and Gharabaghi combined RL with Bayesian model to select dynamic thresholds for improved performance of restorative BMI [183].…”

Section: /33mentioning

confidence: 99%

Applications of Deep Learning and Reinforcement Learning to Biological Data

Mahmud

Kaiser

Hussain

et al. 2018

IEEE Trans. Neural Netw. Learning Syst.

711

283

View full text Add to dashboard Cite

Rapid advances in hardware-based technologies during the past decades have opened up new possibilities for life scientists to gather multimodal data in various application domains, such as omics, bioimaging, medical imaging, and (brain/body)-machine interfaces. These have generated novel opportunities for development of dedicated data-intensive machine learning techniques. In particular, recent research in deep learning (DL), reinforcement learning (RL), and their combination (deep RL) promise to revolutionize the future of artificial intelligence. The growth in computational power accompanied by faster and increased data storage, and declining computing costs have already allowed scientists in various fields to apply these techniques on data sets that were previously intractable owing to their size and complexity. This paper provides a comprehensive survey on the application of DL, RL, and deep RL techniques in mining biological data. In addition, we compare the performances of DL techniques when applied to different data sets across various application domains. Finally, we outline open issues in this challenging research area and discuss future development perspectives.

show abstract

“…As mentioned above, the differences seen in rewarding versus nonrewarding trials through various measures have potential uses in improved learning for BMIs. Toward this goal, our lab recently showed that integrated features of PSD and SFC yielded nearperfect classification accuracy between rewarding and nonrewarding trials during color-cued one-target center-out reaching tasks and thus can be used as a neural critic in autonomous BMI decoding (Bae et al, 2011;Sanchez et al, 2011;Tarigoppula et al, 2012;Marsh et al, 2015;An et al, 2018;Tarigoppula et al, 2018).…”

Section: Alpha Cycle Relation To Neural Spikingmentioning

confidence: 99%

Reward Modulates Local Field Potentials, Spiking Activity and Spike-Field Coherence in the Primary Motor Cortex

Yadav

Hessburg

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Number of words (Abstract): Number of words (Significance Statement): Number of words (Introduction): Number of words (Discussion):Conflict of Interest: The authors declare no conflict of interests. ABSTRACTReward modulation of the primary motor cortex (M1) could be exploited in developing an autonomously updating brain-machine interface (BMI) based on a reinforcement learning architecture. In order to understand the multifaceted effects of reward on M1 activity, we investigated how neural spiking, oscillatory activities and their functional interactions are modulated by conditioned stimuli related reward expectation. To do so, local field potentials (LFPs) and singleunit/multi-unit activities were recorded simultaneously and bilaterally from M1 cortices while five non-human primates performed cued center-out reaching or grip force tasks either manually using their right arm/hand or observed passively.We found that reward expectation influenced the strength of alpha (8)(9)(10)(11)(12)(13)(14) power, alpha-gamma comodulation, alpha spike-field coherence, and firing rates in general in M1. Furthermore, we found that an increase in alpha-band power was correlated with a decrease in neural spiking activity, that firing rates were highest at the trough of the alpha-band cycle and lowest at the peak of its cycle.These findings imply that alpha oscillations modulated by reward expectation have an influence on spike firing rate and spike timing during both reaching and grasping tasks in M1. These LFP, spike, and spike-field interactions could be used to follow the M1 neural state in order to enhance BMI decoding Zhao et al., 2018). Significance StatementKnowing the subjective value of performed or observed actions is valuable feedback that can be used to improve the performance of an autonomously updating brain-machine interface (BMI). Reward-related information in the primary motor cortex (M1) may be crucial for more stable and robust BMI decoding (Zhao et al., 2018). Here, we present how expectation of reward during motor tasks, or simple observation, is represented by increased spike firing rates in conjunction with decreased alpha (8-14 Hz) oscillatory power, alpha-gamma comodulation, and alpha spike-field coherence, as compared to non-rewarding trials. Moreover, a phasic relation between alpha oscillations and firing rates was observed where firing rates were found to be lowest and highest at the peak and trough of alpha oscillations, respectively.

show abstract

Control of a center-out reaching task using a reinforcement learning Brain-Machine Interface

Cited by 34 publications

References 12 publications

Kernel Temporal Differences for Neural Decoding

Kernel Temporal Differences for Neural Decoding

Applications of Deep Learning and Reinforcement Learning to Biological Data

Reward Modulates Local Field Potentials, Spiking Activity and Spike-Field Coherence in the Primary Motor Cortex

Contact Info

Product

Resources

About