Motor cortical activity changes during neuroprosthetic-controlled object interaction

Downey, John E.; Brane, Lucas; Gaunt, Robert A.; Tyler-Kabara, Elizabeth C.; Boninger, Michael L.; Collinger, Jennifer L.

doi:10.1038/s41598-017-17222-3

Cited by 57 publications

(41 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is an important step towards neural speech prostheses for speech impaired users. While our study uses overtly produced speech in non-paralyzed subjects, recent studis show analogous results for paralyzed patients [54,55,56,57], as well as for speech decoding [58], providing reason to believe that similiar results might also be achievable for mute users.…”

Section: Discussionsupporting

confidence: 52%

Speech Synthesis from ECoG using Densely Connected 3D Convolutional Neural Networks

Angrick

Mugler²,

Tate³

et al. 2018

Preprint

View full text Add to dashboard Cite

Objective. Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech. Approach. Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is well-suited to work with the small amount of data available from each participant. Main results. In a study with six participants, we achieved correlations up to r = 0.69 between the reconstructed and original logMel spectrograms. We transfered our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel features that harnessed a much larger, pre-existing data corpus to provide the most natural acoustic output. Significance. To the best of our knowledge, this is the first time that high-quality speech has been reconstructed from neural recordings during speech production using deep neural networks.

show abstract

Section: Discussionsupporting

confidence: 52%

Speech Synthesis from ECoG using Densely Connected 3D Convolutional Neural Networks

Angrick

Mugler²,

Tate³

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Consistent with this recommendation, the Cursor Position Subtraction operation was not used in our recent study in which the ReFIT decoder enabled people with paralysis to type using a virtual keyboard 2 . These results add to an emerging story that the effects of some forms of visual feedback on motor cortical activity may be less problematic for BMI decoding than originally predicted 27 (but see 34 , which reports that seeing a BMI arm approaching a graspable object strongly modulates activity).…”

Section: Discussionmentioning

confidence: 50%

“…Although we showed here that visually monitored BMI effector positional effects are minimal and can be ignored during decoding, it is unknown whether more anthropomorphic BMI effectors such as robotic arms 34,37 will induce stronger motor cortical modulation related to effector state (e.g., arm position or joints conformation). This might be expected because these prostheses more directly tap into the sensorimotor system’s native arm-monitoring machinery.…”

Section: Discussionmentioning

confidence: 77%

Brain-machine interface cursor position only weakly affects monkey and human motor cortical activity in the absence of arm movements

Stavisky

Kao

Nuyujukian

et al. 2018

Sci Rep

View full text Add to dashboard Cite

Brain-machine interfaces (BMIs) that decode movement intentions should ignore neural modulation sources distinct from the intended command. However, neurophysiology and control theory suggest that motor cortex reflects the motor effector’s position, which could be a nuisance variable. We investigated motor cortical correlates of BMI cursor position with or without concurrent arm movement. We show in two monkeys that subtracting away estimated neural correlates of position improves online BMI performance only if the animals were allowed to move their arm. To understand why, we compared the neural variance attributable to cursor position when the same task was performed using arm reaching, versus arms-restrained BMI use. Firing rates correlated with both BMI cursor and hand positions, but hand positional effects were greater. To examine whether BMI position influences decoding in people with paralysis, we analyzed data from two intracortical BMI clinical trial participants and performed an online decoder comparison in one participant. We found only small motor cortical correlates, which did not affect performance. These results suggest that arm movement and proprioception are the major contributors to position-related motor cortical correlates. Cursor position visual feedback is therefore unlikely to affect the performance of BMI-driven prosthetic systems being developed for people with paralysis.

show abstract

“…Recently our lab and others have shown that context can modulate the primary motor cortex (M1) (Downey et al 2017, Marsh et al 2015, Ramakrishnan et al 2017, Ramkumar et al 2016. For instance, it has become clear that a reward signal exists in the primary motor cortex (M1), and primary somatosensory cortex (S1) as well (McNiel et al 2016a, McNiel et al 2016b.…”

Section: Introductionmentioning

confidence: 99%

Paradigm Shift in Sensorimotor Control Research and Brain Machine Interface Control: The Influence of Context on Sensorimotor representations

Zhao

Hessburg

Kumar

et al. 2017

Preprint

View full text Add to dashboard Cite

To be submitted to a special issue on Paradigm shifts. AbstractNeural activity in the primary motor cortex (M1) is known to correlate with movement related variables including kinematics and dynamics. Our recent work, which we believe will lead to a paradigm shift in sensorimotor research, has shown that in addition to these movement related variables, activity in M1, and the primary somatosensory cortex (S1), are also modulated by context, such as value, during both active movement and movement observation. Here we expand on the investigation of reward modulation in M1, showing that reward level changes the neural tuning function of M1 units to both kinematic as well as dynamic related variables. In addition, we show that this reward-modulated activity is present during brain machine interface (BMI) control. We suggest that by taking into account these context dependencies of M1 modulation, we can produce more robust BMIs. Toward this goal, we demonstrate that we can classify reward expectation from M1 on a movement-by-movement basis under BMI control and use this to gate multiple linear BMI decoders toward improved offline performance. These findings demonstrate that it is possible and meaningful to design a more accurate BMI decoder that takes reward and context into consideration. Our next step in this development will be to incorporate this gating system, or a continuous variant of it, into online BMI performance

show abstract

Motor cortical activity changes during neuroprosthetic-controlled object interaction

Cited by 57 publications

References 48 publications

Speech Synthesis from ECoG using Densely Connected 3D Convolutional Neural Networks

Speech Synthesis from ECoG using Densely Connected 3D Convolutional Neural Networks

Brain-machine interface cursor position only weakly affects monkey and human motor cortical activity in the absence of arm movements

Paradigm Shift in Sensorimotor Control Research and Brain Machine Interface Control: The Influence of Context on Sensorimotor representations

Contact Info

Product

Resources

About