A collaborative BCI approach to autonomous control of a prosthetic limb system

Katyal, Kapil D.; Johannes, Matthew S.; Kellis, Spencer; Aflalo, Tyson; Klaes, Christian; McGee, T. G.; Para, Matthew P.; Shi, Ying; Lee, Brian C.; Pejsa, Kelsie; Liu, Charles; Wester, Brock A.; Beaty, James D.; Ravitz, Alan; Andersen, Richard A.; McLoughlin, Michael P.

doi:10.1109/smc.2014.6974124

Cited by 44 publications

(26 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The value of computer vision in BCI controlled manipulation was demonstrated with a simple sphere-and-cylinder detection system [29,15]. We extend upon these findings by utilizing a more advanced perception pipeline (Section II-A) in conjunction with a model library consisting of 3D object models and corresponding pre-labeled grasp sets.…”

Section: Introductionmentioning

confidence: 81%

“…We extend upon these findings by utilizing a more advanced perception pipeline (Section II-A) in conjunction with a model library consisting of 3D object models and corresponding pre-labeled grasp sets. In contrast to the fixed distance based threshold in [15], we introduce capture envelopes, similar to gravity fields [40], for smooth and continuous user grasp inference that can vary based on the object and grasp (Section II-B). Utilizing prior work in value function based user intent inference (Section II-C), we circumvent the requirement of explicit user goal selection by inferring the user's desired goal, similar in motivation to eye tracking and other interfaces [29].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Autonomy Infused Teleoperation with Application to BCI Manipulation

Mülling¹,

Venkatraman

Valois

et al. 2015

Robotics: Science and Systems XI

View full text Add to dashboard Cite

Abstract-Robot teleoperation systems face a common set of challenges including latency, low-dimensional user commands, and asymmetric control inputs. User control with BrainComputer Interfaces (BCIs) exacerbates these problems through especially noisy and erratic low-dimensional motion commands due to the difficulty in decoding neural activity. We introduce a general framework to address these challenges through a combination of computer vision, user intent inference, and arbitration between the human input and autonomous control schemes. Adjustable levels of assistance allow the system to balance the operator's capabilities and feelings of comfort and control while compensating for a task's difficulty. We present experimental results demonstrating significant performance improvement using the shared-control assistance framework on adapted rehabilitation benchmarks with two subjects implanted with intracortical brain-computer interfaces controlling a seven degree-of-freedom robotic manipulator as a prosthetic. Our results further indicate that shared assistance mitigates perceived user difficulty and even enables successful performance on previously infeasible tasks. We showcase the extensibility of our architecture with applications to quality-of-life tasks such as opening a door, pouring liquids from containers, and manipulation with novel objects in densely cluttered environments.

show abstract

Section: Introductionmentioning

confidence: 81%

Section: Introductionmentioning

confidence: 99%

Autonomy Infused Teleoperation with Application to BCI Manipulation

Mülling¹,

Venkatraman

Valois

et al. 2015

Robotics: Science and Systems XI

View full text Add to dashboard Cite

show abstract

“…These preliminary results suggest the possibilities and the advantages of using ROS in BCI driven telepresence applications. The proposed BCI system is one of the few working on the top of a ROS framework [9], [10], [11] and, among them, the only one supporting an endogenous SMR based BCI. As in the case of previous works [4], the designed semi-autonomous control reduced the user's fatigue (in terms of number of commands required to reach the target).…”

Section: Discussionmentioning

confidence: 99%

Brain-Computer Interface Meets ROS: A Robotic Approach to Mentally Drive Telepresence Robots

Beraldo

Antonello

Cimolato

et al. 2018

2018 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

This paper shows and evaluates a novel approach to integrate a non-invasive Brain-Computer Interface (BCI) with the Robot Operating System (ROS) to mentally drive a telepresence robot. Controlling a mobile device by using human brain signals might improve the quality of life of people suffering from severe physical disabilities or elderly people who cannot move anymore. Thus, the BCI user is able to actively interact with relatives and friends located in different rooms thanks to a video streaming connection to the robot. To facilitate the control of the robot via BCI, we explore new ROS-based algorithms for navigation and obstacle avoidance, making the system safer and more reliable. In this regard, the robot can exploit two maps of the environment, one for localization and one for navigation, and both can be used also by the BCI user to watch the position of the robot while it is moving. As demonstrated by the experimental results, the user's cognitive workload is reduced, decreasing the number of commands necessary to complete the task and helping him/her to keep attention for longer periods of time.

show abstract

“…The immense utility of incorporating continuous shared control with environmental sensors has been shown online with human subjects [9], [24]. By using environmental sensors for object localization, the subjects in [9] were able to complete tasks consistently that could not be completed as well with neural control alone.…”

Section: Discussionmentioning

confidence: 99%

High Precision Neural Decoding of Complex Movement Trajectories Using Recursive Bayesian Estimation With Dynamic Movement Primitives

Hotson

Smith

Rouse

et al. 2016

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Brain-machine interfaces (BMIs) are a rapidly progressing technology with the potential to restore function to victims of severe paralysis via neural control of robotic systems. Great strides have been made in directly mapping a user's cortical activity to control of the individual degrees of freedom of robotic end-effectors. While BMIs have yet to achieve the level of reliability desired for widespread clinical use, environmental sensors (e.g. RGB-D cameras for object detection) and prior knowledge of common movement trajectories hold great potential for improving system performance. Here we present a novel sensor fusion paradigm for BMIs that capitalizes on information able to be extracted from the environment to greatly improve the performance of control. This was accomplished by using dynamic movement primitives to model the 3D endpoint trajectories of manipulating various objects. We then used a switching unscented Kalman filter to continuously arbitrate between the 3D endpoint kinematics predicted by the dynamic movement primitives and control derived from neural signals. We experimentally validated our system by decoding 3D endpoint trajectories executed by a non-human primate manipulating four different objects at various locations. Performance using our system showed a dramatic improvement over using neural signals alone, with median distance between actual and decoded trajectories decreasing from 31.1 cm to 9.9 cm, and mean correlation increasing from 0.80 to 0.98. Our results indicate that our sensor fusion framework can dramatically increase the fidelity of neural prosthetic trajectory decoding.

show abstract

A collaborative BCI approach to autonomous control of a prosthetic limb system

Cited by 44 publications

References 19 publications

Autonomy Infused Teleoperation with Application to BCI Manipulation

Autonomy Infused Teleoperation with Application to BCI Manipulation

Brain-Computer Interface Meets ROS: A Robotic Approach to Mentally Drive Telepresence Robots

High Precision Neural Decoding of Complex Movement Trajectories Using Recursive Bayesian Estimation With Dynamic Movement Primitives

Contact Info

Product

Resources

About