Controlling Assistive Robots with Learned Latent Actions

Losey, Dylan P.; Srinivasan, K.; Mandlekar, Ajay; Garg, Animesh; Sadigh, Dorsa

doi:10.1109/icra40945.2020.9197197

Cited by 48 publications

(34 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, in [2], tetraplegic users can use an eye tracker for indicating the desired object through their gaze, which makes the robot automatically follow an offline planned path towards the object. Other approaches, like [3], grant more autonomy to users with more mobility and allow them to control a 7 DoF manipulator in a reduced learned latent space controlled by a 2D joystick during object reaching tasks. However, the authors conclude that the latent actions became counter-intuitive and erratic when the state of the robot is not near the training data.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Reconfigurable Constraint-Based Reactive Framework for Assistive Robotics With Adaptable Levels of Autonomy

Iregui

Schutter

Aertbeliën

2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

“…The orientation path can be computed as R(s) = R 0 exp([ n×]f θ (s)) with 0 ≤ s ≤ 1. A smooth evolution of the angle f θ that ensures zero derivatives at s = 0 and s = 1 can be given, for example, by the cubic polynomial f θ (s) = 3 θs 2 − 2 θs 3 .…”

Section: Automatic Orientation Path (Ram)mentioning

confidence: 99%

Reconfigurable Constraint-Based Reactive Framework for Assistive Robotics With Adaptable Levels of Autonomy

Iregui

Schutter

Aertbeliën

2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

“…Building on these ideas, researchers have proposed learning SC policies directly from demonstration data using deep reinforcement learning (Reddy et al, 2018). To improve the human partner’s intuition for the interaction paradigm, researchers have also proposed learning latent spaces to allow users to control complex robots with low-dimensional input devices (Losey et al, 2019). Relatedly, people have also proposed techniques for modeling both the dynamics of a system, and a policy for deciding when a human or autonomous partner should be in control.…”

Section: Background and Related Workmentioning

confidence: 99%

Data-driven Koopman operators for model-based shared control of human–machine systems

Broad

Abraham

Murphey

et al. 2020

The International Journal of Robotics Research

View full text Add to dashboard Cite

We present a data-driven shared control algorithm that can be used to improve a human operator’s control of complex dynamic machines and achieve tasks that would otherwise be challenging, or impossible, for the user on their own. Our method assumes no a priori knowledge of the system dynamics. Instead, both the dynamics and information about the user’s interaction are learned from observation through the use of a Koopman operator. Using the learned model, we define an optimization problem to compute the autonomous partner’s control policy. Finally, we dynamically allocate control authority to each partner based on a comparison of the user input and the autonomously generated control. We refer to this idea as model-based shared control (MbSC). We evaluate the efficacy of our approach with two human subjects studies consisting of 32 total participants (16 subjects in each study). The first study imposes a linear constraint on the modeling and autonomous policy generation algorithms. The second study explores the more general, nonlinear variant. Overall, we find that MbSC significantly improves task and control metrics when compared with a natural learning, or user only, control paradigm. Our experiments suggest that models learned via the Koopman operator generalize across users, indicating that it is not necessary to collect data from each individual user before providing assistance with MbSC. We also demonstrate the data efficiency of MbSC and, consequently, its usefulness in online learning paradigms. Finally, we find that the nonlinear variant has a greater impact on a user’s ability to successfully achieve a defined task than the linear variant.

show abstract

“…However, the direct teleoperation of robotic systems is challenging and often places significant cognitive burden on the operator ( Tanwani and Calinon, 2017 ; Xi et al, 2019 ; Hetrick et al, 2020 ). This is especially true when handling robots with high degrees-of-freedom, like robotic arms for grasping and manipulation ( Losey et al, 2020 ). In order to alleviate any excess workload exerted on a teleoperator, shared control is typically employed as a means of providing autonomous assistance.…”

Section: Introductionmentioning

confidence: 99%

Motion Polytopes in Virtual Reality for Shared Control in Remote Manipulation Applications

et al. 2021

View full text Add to dashboard Cite

In remote applications that mandate human supervision, shared control can prove vital by establishing a harmonious balance between the high-level cognition of a user and the low-level autonomy of a robot. Though in practice, achieving this balance is a challenging endeavor that largely depends on whether the operator effectively interprets the underlying shared control. Inspired by recent works on using immersive technologies to expose the internal shared control, we develop a virtual reality system to visually guide human-in-the-loop manipulation. Our implementation of shared control teleoperation employs end effector manipulability polytopes, which are geometrical constructs that embed joint limit and environmental constraints. These constructs capture a holistic view of the constrained manipulator’s motion and can thus be visually represented as feedback for users on their operable space of movement. To assess the efficacy of our proposed approach, we consider a teleoperation task where users manipulate a screwdriver attached to a robotic arm’s end effector. A pilot study with prospective operators is first conducted to discern which graphical cues and virtual reality setup are most preferable. Feedback from this study informs the final design of our virtual reality system, which is subsequently evaluated in the actual screwdriver teleoperation experiment. Our experimental findings support the utility of using polytopes for shared control teleoperation, but hint at the need for longer-term studies to garner their full benefits as virtual guides.

show abstract

Controlling Assistive Robots with Learned Latent Actions

Cited by 48 publications

References 16 publications

Reconfigurable Constraint-Based Reactive Framework for Assistive Robotics With Adaptable Levels of Autonomy

Reconfigurable Constraint-Based Reactive Framework for Assistive Robotics With Adaptable Levels of Autonomy

Data-driven Koopman operators for model-based shared control of human–machine systems

Motion Polytopes in Virtual Reality for Shared Control in Remote Manipulation Applications

Contact Info

Product

Resources

About