HARMONIC: A Multimodal Dataset of Assistive Human-Robot Collaboration

Newman, Benjamin A.; Aronson, Reuben M.; Srinivasa, Siddartha S.; Admoni, Henny

doi:10.48550/arxiv.1807.11154

Cited by 5 publications

(15 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Latent Actions. Prior work on shared autonomy assumes a pre-defined teleoperation mapping with multiple modes [7], [28]. Consider using a 2-DoF joystick to control a robot arm: your joystick moves the robot's end-effector along the x-y axes in one mode, in z-roll axes in another mode, and so on.…”

Section: Problem Statementmentioning

confidence: 99%

“…Within End-Effector participants directly controlled the velocity of the robot's end-effector. They pressed a button to toggle between two different modes: one mode controlled the robot's linear velocity, and the other controlled the robot's angular velocity [7], [28]. By contrast, with Ours the robot mapped the user's 2-DoF joystick input to joint velocities.…”

Section: User Studymentioning

confidence: 99%

See 1 more Smart Citation

Learning Latent Actions without Human Demonstrations

Mehta¹,

Parekh²,

Losey³

2021

Preprint

View full text Add to dashboard Cite

We can make it easier for disabled users to control assistive robots by mapping the user's low-dimensional joystick inputs to high-dimensional, complex actions. Prior works learn these mappings from human demonstrations: a non-disabled human either teleoperates or kinesthetically guides the robot arm through a variety of motions, and the robot learns to reproduce the demonstrated behaviors. But this framework is often impractical -disabled users will not always have access to external demonstrations! Here we instead learn diverse teleoperation mappings without either human demonstrations or predefined tasks. Under our unsupervised approach the robot first optimizes for object state entropy: i.e., the robot autonomously learns to push, pull, open, close, or otherwise change the state of nearby objects. We then embed these diverse, object-oriented behaviors into a latent space for real-time control: now pressing the joystick causes the robot to perform dexterous motions like pushing or opening. We experimentally show that -with a best-case human operator -our unsupervised approach actually outperforms the teleoperation mappings learned from human demonstrations, particularly if those demonstrations are noisy or imperfect. But user study results are less clear-cut: although our approach enables participants to complete tasks with multiple objects more quickly, the unsupervised mapping also learns motions that the human does not need, and these additional behaviors may confuse the human. Videos of the user study: https://youtu.be/BkqHQjsUKDg

show abstract

Section: Problem Statementmentioning

confidence: 99%

Section: User Studymentioning

confidence: 99%

Learning Latent Actions without Human Demonstrations

Mehta¹,

Parekh²,

Losey³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Prior work on assistive arms leverages shared autonomy, where the robot's action is a combination of the human's input and autonomous assistance [16,25,24,7]. Here the human controls the robot with a low-DoF interface (typically a joystick), and the robot leverages a pre-defined mapping with toggled modes to convert the human's inputs into end-effector motion [3,22,37]. To assist the human, the robot maintains a belief over a discrete set of possible goal objects in the environment: the robot continually updates this belief by leveraging the human's joystick inputs as evidence in a Bayesian framework [16,25,24,20,38].…”

Section: Shared Autonomymentioning

confidence: 99%

“…Existing work on assistive robots tackles this problem with pre-defined mappings between user inputs and robot actions. These mappings incorporate modes, and the user switches between modes to control different robot DoFs [22,3,37]. For instance, in one mode the user's 2-DoF joystick controls the x-y position of the end-effector, in a second mode the joystick controls the z-yaw position of the end-effector, and so on.…”

Section: Introductionmentioning

confidence: 99%

“…Non-disabled participants teleoperated a 7-DoF robot arm using a 2-DoF joystick to reach marshmallows, make a simplified apple pie, cut tofu, and assemble dessert. We compared our latent action approach to both pre-defined mappings and shared autonomy baselines, including the HARMONIC dataset [37]. We found that latent actions help users complete high-level reaching and precise manipulations with their preferred alignment, resulting in improved objective and subjective performance.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Latent Actions to Control Assistive Robots

Losey¹,

Jeon²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

Assistive robot arms enable people with disabilities to conduct everyday tasks on their own. These arms are dexterous and high-dimensional ; however, the interfaces people must use to control their robots are low-dimensional. Consider teleoperating a 7-DoF robot arm with a 2-DoF joystick. The robot is helping you eat dinner, and currently you want to cut a piece of tofu. Today's robots assume a pre-defined mapping between joystick inputs and robot actions: in one mode the joystick controls the robot's motion in the x-y plane, in another mode the joystick controls the robot's z-yaw motion, and so on. But this mapping misses out on the task you are trying to perform! Ideally, one joystick axis should control how the robot stabs the tofu, and the other axis should control different cutting motions. Our insight is that we can achieve intuitive, userfriendly control of assistive robots by embedding the robot's high-dimensional actions into low-dimensional and human-controllable latent actions. We divide this process into three parts. First, we explore models for learning latent actions from offline task demonstrations, and formalize the properties that latent actions should satisfy. Next, we combine learned latent actions with autonomous robot assistance to help the user reach and maintain their high-level goals. Finally, we learn a personalized alignment model between joystick inputs and latent actions. We evaluate our resulting approach in four user studies where non-disabled participants reach

show abstract

Core Challenges in Embodied Vision-Language Planning

Francis

Kitamura

Labelle

et al. 2022

jair

View full text Add to dashboard Cite

Recent advances in the areas of multimodal machine learning and artificial intelligence (AI) have led to the development of challenging tasks at the intersection of Computer Vision, Natural Language Processing, and Embodied AI. Whereas many approaches and previous survey pursuits have characterised one or two of these dimensions, there has not been a holistic analysis at the center of all three. Moreover, even when combinations of these topics are considered, more focus is placed on describing, e.g., current architectural methods, as opposed to also illustrating high-level challenges and opportunities for the field. In this survey paper, we discuss Embodied Vision-Language Planning (EVLP) tasks, a family of prominent embodied navigation and manipulation problems that jointly use computer vision and natural language. We propose a taxonomy to unify these tasks and provide an in-depth analysis and comparison of the new and current algorithmic approaches, metrics, simulated environments, as well as the datasets used for EVLP tasks. Finally, we present the core challenges that we believe new EVLP works should seek to address, and we advocate for task construction that enables model generalizability and furthers real-world deployment.

show abstract

HARMONIC: A Multimodal Dataset of Assistive Human-Robot Collaboration

Cited by 5 publications

References 0 publications

Learning Latent Actions without Human Demonstrations

Learning Latent Actions without Human Demonstrations

Learning Latent Actions to Control Assistive Robots

Core Challenges in Embodied Vision-Language Planning

Contact Info

Product

Resources

About