Cortical control of a tablet computer by people with paralysis

Nuyujukian, Paul; Albites-Sanabria, Jose; Saab, Jad; Pandarinath, Chethan; Jarosiewicz, Beata; Blabe, Christine H; Franco, Brian; Mernoff, Stephen T.; Eskandar, Emad N.; Simeral, John D.; Hochberg, Leigh R.; Shenoy, Krishna V.; Henderson, Jaimie M.

doi:10.1371/journal.pone.0204566

Cited by 124 publications

(92 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the last decade, brain-machine interfaces (BMIs) have transitioned from animal models into human subjects, demonstrating that some amount of motor function can be restored to tetraplegics— typically, continuous movements with two degrees of freedom ( Nuyujukian et al, 2018; Gilja et al, 2015; Jarosiewicz et al, 2015 ). Although this type of control can be used in conjunction with a virtual keyboard to produce text, even under ideal cursor control (not currently achievable), the word rate would still be limited to that of typing with a single finger.…”

Section: Introductionmentioning

confidence: 99%

Machine translation of cortical activity to text with an encoder-decoder framework

Makin

Moses

Chang

2019

Preprint

103

View full text Add to dashboard Cite

AbstractA decade after the first successful attempt to decode speech directly from human brain signals, accuracy and speed remain far below that of natural speech or typing. Here we show how to achieve high accuracy from the electrocorticogram at natural-speech rates, even with few data (on the order of half an hour of spoken speech). Taking a cue from recent advances in machine translation and automatic speech recognition, we train a recurrent neural network to map neural signals directly to word sequences (sentences). In particular, the network first encodes a sentence-length sequence of neural activity into an abstract representation, and then decodes this representation, word by word, into an English sentence. For each participant, training data consist of several spoken repeats of a set of some 30-50 sentences, along with the corresponding neural signals at each of about 250 electrodes distributed over peri-Sylvian speech cortices. Average word error rates across a validation (held-out) sentence set are as low as 7% for some participants, as compared to the previous state of the art of greater than 60%. Finally, we show how to use transfer learning to overcome limitations on data availability: Training certain components of the network under multiple participants’ data, while keeping other components (e.g., the first hidden layer) “proprietary,” can improve decoding performance—despite very different electrode coverage across participants.

show abstract

Section: Introductionmentioning

confidence: 99%

Machine translation of cortical activity to text with an encoder-decoder framework

Makin

Moses

Chang

2019

Preprint

103

View full text Add to dashboard Cite

show abstract

“…Research in the area of BCIs is currently evolving [56], with promising results in recent state-of-the-art projects. A study by Stanford University [73] confirmed the usability of BCIs to control an unmodified smart device for quadriplegic users. BCIs have been also in use to surf the internet [74], with an EEG BCI based application tested with LIS and ALS conditions [75].…”

Section: Touch-activated Systemsmentioning

confidence: 93%

Augmentative and Alternative Communication (AAC) Advances: A Review of Configurations for Speech Disabled Individuals

Elsahar¹,

Hu²,

Bouazza-Marouf³

et al. 2019

Preprint

View full text Add to dashboard Cite

High-tech augmentative and alternative communication (AAC) methods are on a constant rise; however, the interaction between the user and the assistive technology is still challenged for an optimal user experience centered around the desired activity. This review presents a range of signal sensing and acquisition methods utilized in conjunction with the existing high-tech AAC platforms for speech disabled individuals, including imaging methods, touch-enabled systems, mechanical and electro-mechanical access, breath-activated methods, and brain computer interfaces (BCI). The listed AAC sensing modalities are compared in terms of ease of access, affordability, complexity, portability, and typical conversational speeds. A revelation of the associated AAC signal processing, encoding, and retrieval highlights the roles of machine learning (ML) and deep learning (DL) in the development of intelligent AAC solutions. The demands and the affordability of most systems were found to hinder the scale of usage of high-tech AAC. Further research is indeed needed for the development of intelligent AAC applications reducing the associated costs and enhancing the portability of the solutions for a real user’s environment. The consolidation of natural language processing with current solutions also needs to be further explored for the amelioration of the conversational speeds. The recommendations for prospective advances in coming high-tech AAC are addressed in terms of developments to support mobile health communicative applications.

show abstract

“…To evaluate whether the wireless neural interface system could enable individuals with paralysis to achieve reliable point-and-click control of a computer in their homes, T5 and T10 used the wireless iBCI for point-and-click control of an unmodified consumer mobile device (Microsoft Surface Pro tablet) for web browsing, use of consumer applications, and typing communication. Cursor control was enabled using our standard automated decoder calibration, then CNRAs placed the tablet comfortably in front of the participant and routed the iBCI decoded cursor and click commands to the tablet [8]. Participants used the tablets in Microsoft Windows 10 "tablet" mode to navigate the desktop and to start and use different programs of interest to them.…”

Section: Wireless Ibci Use Of a Tablet Computermentioning

confidence: 99%

“…Motor BCIs aim to provide access to assistive devices by decoding user commands from electroencephalography (EEG), electrocorticography (ECoG), or intracortical signals. In ongoing clinical research, high-performance intracortical BCIs (iBCIs) are being developed to infer a user's movement intentions [1]- [8] or speech [9], [10] from neural activity recorded from one or more microelectrode arrays implanted in motor areas of cortex. By imagining natural hand, finger and arm movements, trial participants with paralysis have achieved reach-and grasp with robotic and prosthetic limbs [2], [3], [11] and their own reanimated limb [6], [12], and have demonstrated reliable cursor control for tablet use [8] and on-screen typing for communication [4], [7], [13].…”

mentioning

confidence: 99%

“…In ongoing clinical research, high-performance intracortical BCIs (iBCIs) are being developed to infer a user's movement intentions [1]- [8] or speech [9], [10] from neural activity recorded from one or more microelectrode arrays implanted in motor areas of cortex. By imagining natural hand, finger and arm movements, trial participants with paralysis have achieved reach-and grasp with robotic and prosthetic limbs [2], [3], [11] and their own reanimated limb [6], [12], and have demonstrated reliable cursor control for tablet use [8] and on-screen typing for communication [4], [7], [13]. Building on steady advances in point-and-click accuracy, speed [4], [7], [13]- [16] and consistency [17]- [21], iBCI trial participants at home have achieved average on-screen point-to-select typing rates over 37 correct characters per minute maintained over days and weeks [4], [7].…”

mentioning

confidence: 99%

See 1 more Smart Citation

Home Use of a Wireless Intracortical Brain-Computer Interface by Individuals With Tetraplegia

Hosman

Saab

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Individuals with neurological disease or injury such as amyotrophic lateral sclerosis, spinal cord injury or stroke may become tetraplegic, unable to speak or even locked-in. For people with these conditions, current assistive technologies are often ineffective. Brain-computer interfaces are being developed to enhance independence and restore communication in the absence of physical movement. Over the past decade, individuals with tetraplegia have achieved rapid on-screen typing and point-andclick control of tablet apps using intracortical brain-computer interfaces (iBCIs) that decode intended arm and hand movements from neural signals recorded by implanted microelectrode arrays. However, cables used to convey neural signals from the brain tether participants to amplifiers and decoding computers and require expert oversight during use, severely limiting when and where iBCIs could be available for use. Here, we demonstrate the first human use of a wireless broadband iBCI. Based on a prototype system previously used in pre-clinical research, we replaced the external cables of a 192-electrode iBCI with wireless transmitters and achieved high-resolution recording and decoding of broadband field potentials and spiking activity from people with paralysis. Two participants in an ongoing pilot clinical trial performed on-screen item selection tasks to assess iBCI-enabled cursor control. Communication bitrates were equivalent between cabled and wireless configurations. Participants also used the wireless iBCI to control a standard commercial tablet computer to browse the web and use several mobile applications. Within-day comparison of cabled and wireless interfaces evaluated bit error rate, packet loss, and the recovery of spike rates and spike waveforms from the recorded neural signals. In a representative use case, the wireless system recorded intracortical signals from two arrays in one participant continuously through a 24-hour period at home. Wireless multi-electrode recording of broadband neural signals over extended periods introduces a valuable tool for human neuroscience research and is an important step toward practical deployment of iBCI technology for independent use by individuals with paralysis. On-demand access to highperformance iBCI technology in the home promises to enhance independence and restore communication and mobility for individuals with severe motor impairment. 1

show abstract

Cortical control of a tablet computer by people with paralysis

Cited by 124 publications

References 35 publications

Machine translation of cortical activity to text with an encoder-decoder framework

Machine translation of cortical activity to text with an encoder-decoder framework

Augmentative and Alternative Communication (AAC) Advances: A Review of Configurations for Speech Disabled Individuals

Home Use of a Wireless Intracortical Brain-Computer Interface by Individuals With Tetraplegia

Contact Info

Product

Resources

About