Shared Autonomy with Learned Latent Actions

Abstract

Assistive robots enable people with disabilities to conduct everyday tasks on their own. However, these tasks can be complex, containing both coarse reaching motions and fine-grained manipulation. For example, when eating, not only does one need to move to the correct food item, but they must also precisely manipulate the food in different ways (e.g., cutting, stabbing, scooping). Shared autonomy methods make robot teleoperation safer and more precise by arbitrating user inputs with robot controls. However, these works have focused mainly on the high-level task of reaching a goal from a discrete set, while largely ignoring manipulation of objects at that goal. Meanwhile, dimensionality reduction techniques for teleoperation map useful high-dimensional robot actions into an intuitive lowdimensional controller, but it is unclear if these methods can achieve the requisite precision for tasks like eating. Our insight is that-by combining intuitive embeddings from learned latent actions with robotic assistance from shared autonomy-we can enable precise assistive manipulation. In this work, we adopt learned latent actions for shared autonomy by proposing a new model structure that changes the meaning of the human's input based on the robot's confidence of the goal. We show convergence bounds on the robot's distance to the most likely goal, and develop a training procedure to learn a controller that is able to move between goals even in the presence of shared autonomy. We evaluate our method in simulations and an eating user study. See videos of our experiments here: https://youtu.be/7BouKojzVyk.

Core Challenges in Embodied Vision-Language Planning

“…In order for EVLP agents to perform complex tasks that require dynamic interaction with other agents, the underlying simulation environments and task structures must support the desired interaction settings. Primarily, environments must enable the representation of others' anticipated actions, mental state, and previous behaviour, which has been shown to be critical in related areas, such as social navigation (Tsai and Oh, 2020;Vemula et al, 2018;Mavrogiannis et al, 2021), natural language processing (Fried et al, 2018a(Fried et al, , 2021Zhu et al, 2021a), and human-machine in-teraction (Newman et al, 2018;Jeon et al, 2020;Charalampous et al, 2017). We highlight these related fields as inspiration for this new direction in EVLP research.…”

Human-AI Shared Control via Policy Dissection

“…A series of works study this problem based on the Atari Game [48,52,8,31]. In addition, the human-AI shared control system is built on autonomous vehicle [19], robotics arm [27] and in multi-agent setting [31]. The reported results on various tasks reflect the effectiveness and efficiency of training and coordination with human-assistive AI.…”

Semi-Autonomous Teleoperation via Learning Non-Prehensile Manipulation Skills

“…While the method proposed in [2] effectively learns a low dimensional latent action space and provides a low dimensional control interface, e.g., 1-DoF latent action, however, the method excessively restricts the freedom of user where the participants do not clearly prefer the proposed method over the conventional teleoperation method due to the limited freedom in control. In [3], Jeon et al have focused Fig. 1: An overview of the proposed teleoperation framework where the blue colored boxes and arrow indicate the user's command.…”