Autonomy Infused Teleoperation with Application to BCI Manipulation

Mülling, Katharina; Venkatraman, Arun; Valois, Jean-Sebastien; Downey, John E.; Weiss, Jeffrey M; Javdani, Shervin; Hebert, Martial; Schwartz, Andrew B.; Collinger, Jennifer L.; Bagnell, J. Andrew

doi:10.15607/rss.2015.xi.039

Cited by 42 publications

(44 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cognitive neuroscience and systems neurophysiology are converging on the idea of movement as communication of intention from the brain to the external world. A revolution is occurring as machines are being designed to infer and operate on intention and to predict future action (Levinson et al, 2011; Muelling et al, 2015; Stentz et al, 2015). In biological systems, these factors are evident as the incorporation of expectation in even the simplest neuronal operations.…”

Section: Discussionmentioning

confidence: 99%

Movement: How the Brain Communicates with the World

Schwartz

2016

Cell

105

View full text Add to dashboard Cite

Voluntary movement is a result of signals transmitted through a communication channel that links the internal world in our minds to the physical world around us. Intention can be considered the desire to effect change on our environment, and this is contained in the signals from the brain, passed through the nervous system to converge on muscles that generate displacements and forces on our surroundings. The resulting changes in the world act to generate sensations that feed back to the nervous system, closing the control loop. This Perspective discusses the experimental and theoretical underpinnings of current models of movement generation and the way they are modulated by external information. Movement systems embody intentionality and prediction, two factors that are propelling a revolution in engineering. Development of movement models that include the complexities of the external world may allow a better understanding of the neuronal populations regulating these processes, as well as the development of solutions for autonomous vehicles and robots, and neural prostheses for those who are motor impaired.

show abstract

Section: Discussionmentioning

confidence: 99%

Movement: How the Brain Communicates with the World

Schwartz

2016

Cell

105

View full text Add to dashboard Cite

show abstract

“…The immense utility of incorporating continuous shared control with environmental sensors has been shown online with human subjects [9], [24]. By using environmental sensors for object localization, the subjects in [9] were able to complete tasks consistently that could not be completed as well with neural control alone.…”

Section: Discussionmentioning

confidence: 99%

“…By using environmental sensors for object localization, the subjects in [9] were able to complete tasks consistently that could not be completed as well with neural control alone. Here we built on this by incorporating DMPs with Bayesian inference to track the user's current action and handle the challenge of arbitration between user commands and the nonlinear dynamics of autonomous robot control.…”

Section: Discussionmentioning

confidence: 99%

“…It can also involve segmentation of user inputs with a hidden Markov model (HMM) [4] in order to draw from a library of movement primitives that aid in task completion [5], [6]. Arbitration between the system predictions and user commands in teleoperated robotics is often accomplished with a continuous linear blending of user inputs with system predictions [3], [7], [8], which has been used to deliver shared control of a BMI [9]. Intelligent arbitration is paramount to user satisfaction [3] with teleoperated robotics, and is integral for the success of shared control with BMIs [10].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

High Precision Neural Decoding of Complex Movement Trajectories Using Recursive Bayesian Estimation With Dynamic Movement Primitives

Hotson

Smith

Rouse

et al. 2016

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Brain-machine interfaces (BMIs) are a rapidly progressing technology with the potential to restore function to victims of severe paralysis via neural control of robotic systems. Great strides have been made in directly mapping a user's cortical activity to control of the individual degrees of freedom of robotic end-effectors. While BMIs have yet to achieve the level of reliability desired for widespread clinical use, environmental sensors (e.g. RGB-D cameras for object detection) and prior knowledge of common movement trajectories hold great potential for improving system performance. Here we present a novel sensor fusion paradigm for BMIs that capitalizes on information able to be extracted from the environment to greatly improve the performance of control. This was accomplished by using dynamic movement primitives to model the 3D endpoint trajectories of manipulating various objects. We then used a switching unscented Kalman filter to continuously arbitrate between the 3D endpoint kinematics predicted by the dynamic movement primitives and control derived from neural signals. We experimentally validated our system by decoding 3D endpoint trajectories executed by a non-human primate manipulating four different objects at various locations. Performance using our system showed a dramatic improvement over using neural signals alone, with median distance between actual and decoded trajectories decreasing from 31.1 cm to 9.9 cm, and mean correlation increasing from 0.80 to 0.98. Our results indicate that our sensor fusion framework can dramatically increase the fidelity of neural prosthetic trajectory decoding.

show abstract

“…A special case of shared decision making is shared control: fuse human and robot platform commands. Examples of shared control abound; assistive wheelchair technology [31], braincomputer interface telemanipulation [43], surgical robots [45], search and rescue [9], extraterrestrial robotics [6], and assistive automobile driving [3] are just a few examples. For any of these cases, shared control can be broken down into a human and robot modeling step, a human and robot prediction step, and an arbitration step over the human and robot predictions (see Section II and [23] for justification of this schema).…”

Section: Introductionmentioning

confidence: 99%

Breaking the Human-Robot Deadlock: Surpassing Shared Control Performance Limits with Sparse Human-Robot Interaction

Trautman

2017

Robotics: Science and Systems XIII

View full text Add to dashboard Cite

Abstract-Human machine teaming has, for decades, been conceptualized as a function allocation (FA) or levels of autonomy (LOA) process: the human is suited for some tasks, while the machine is suitable for others, and as machines improve they take over duties previously assigned to humans. A wide variety of methods-including adaptive, adjustable, blended, supervisory and mixed initiative control, implemented discretely or continuously, as potential fields, as virtual fixture interfaces, or haptic interfaces-are derivatives of FA/LOA. We formalize FA/LOA (and all their derivatives) under a single mathematical formulation called classical shared control (CSC). Despite the widespread adoption of CSC, we prove that it fails to optimize human and robot agreement and intent if either the human or robot model displays "intention ambiguity" (e.g., the human's intended goal is unclear or the robot finds multiple viable solutions). Practically, this suboptimality can manifest as unnecessary and unresolvable disagreement (an unnecessary deadlock). For instance, if the robot chooses to go left around an obstacle and the human chooses to go right, CSC only provides two solutions: freeze in place or collide with the obstacle (we provide a wide variety of failure examples in [52], https://arxiv.org/abs/1611.09490). We find that CSC suboptimality stems from arbitrating over model samples, rather than over models. Our key insight is thus to arbitrate over human and robot distributions; we prove this method optimizes human and robot agreement and intent and resolves deadlocking. Our key contribution is computationally efficient distribution arbitration: if the human and robot carry N our joint has fewer modes than the individual agent models. We call our approach N min -sparse generalized shared control.

show abstract

Autonomy Infused Teleoperation with Application to BCI Manipulation

Cited by 42 publications

References 51 publications

Movement: How the Brain Communicates with the World

Movement: How the Brain Communicates with the World

High Precision Neural Decoding of Complex Movement Trajectories Using Recursive Bayesian Estimation With Dynamic Movement Primitives

Breaking the Human-Robot Deadlock: Surpassing Shared Control Performance Limits with Sparse Human-Robot Interaction

Contact Info

Product

Resources

About