Deep Episodic Memory: Encoding, Recalling, and Predicting Episodic Experiences for Robot Action Execution

Rothfuss, Jonas; Ferreira, Fábio; Aksoy, Eren Erdal; Zhou, You; Asfour, Tamim

doi:10.1109/lra.2018.2860057

Cited by 32 publications

(13 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The majority of work in behaviour cloning operates on a set of configuration-space trajectories that can be collected via tele-operation [15], [16], kinesthetic teaching [17], [18], sensors on a human demonstrator [19], [20], [21], [22], through motion planners [5], or even by observing humans directly. Expanding further on the latter, learning by observing humans has previously been achieved through hand-designed mappings between human actions and robot actions [1], [2], [23], visual activity recognition and explicit handtracking [24], [25], and more recently by a system that infers actions from a single video of a human via an end-toend trained system [4].…”

Section: Related Workmentioning

confidence: 99%

Learning One-Shot Imitation From Humans Without Humans

Bonardi

James

Davison

2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Humans can naturally learn to execute a new task by seeing it performed by other individuals once, and then reproduce it in a variety of configurations. Endowing robots with this ability of imitating humans from third person is a very immediate and natural way of teaching new tasks. Only recently, through meta-learning, there have been successful attempts to one-shot imitation learning from humans; however, these approaches require a lot of human resources to collect the data in the real world to train the robot. But is there a way to remove the need for real world human demonstrations during training? We show that with Task-Embedded Control Networks, we can infer control polices by embedding human demonstrations that can condition a control policy and achieve one-shot imitation learning. Importantly, we do not use a real human arm to supply demonstrations during training, but instead leverage domain randomisation in an application that has not been seen before: sim-to-real transfer on humans. Upon evaluating our approach on pushing and placing tasks in both simulation and in the real world, we show that in comparison to a system that was trained on real-world data we are able to achieve similar results by utilising only simulation data. Videos can be found here * .

show abstract

Section: Related Workmentioning

confidence: 99%

Learning One-Shot Imitation From Humans Without Humans

Bonardi

James

Davison

2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…This is not a simple question as decisions must be made about which students, if any, to prioritize: weaker students who require help to meet minimum learning requirements, or stronger students who are trying hard to learn. Additionally, another attractive prospect would be if a robot can itself learn to improve its knowledge base, e.g., from YouTube videos [75]; domain-specific learning has been demonstrated in various studies but general learning across various topics, and machine creativity for generating new content, are still desirable goals for future work.…”

Section: Future Workmentioning

confidence: 99%

Using the Engagement Profile to Design an Engaging Robotic Teaching Assistant for Students

Cooney

Leister

2019

Robotics

View full text Add to dashboard Cite

We report on an exploratory study conducted at a graduate school in Sweden with a humanoid robot, Baxter. First, we describe a list of potentially useful capabilities for a robot teaching assistant derived from brainstorming and interviews with faculty members, teachers, and students. These capabilities consist of reading educational materials out loud, greeting, alerting, allowing remote operation, providing clarifications, and moving to carry out physical tasks. Secondly, we present feedback on how the robot’s capabilities, demonstrated in part with the Wizard of Oz approach, were perceived, and iteratively adapted over the course of several lectures, using the Engagement Profile tool. Thirdly, we discuss observations regarding the capabilities and the development process. Our findings suggest that using a social robot as a teaching assistant is promising using the chosen capabilities and Engagement Profile tool. We find that enhancing the robot’s autonomous capabilities and further investigating the role of embodiment are some important topics to be considered in future work.

show abstract

“…We focus on this problem of learning from one video demonstration of a human performing a task, in combination with human and robot demonstration data collected on other tasks. Prior work has proposed to resolve the correspondence problem by hand, for example, by manually specifying how human grasp poses correspond to robot grasps [20] or by manually defining how human activities or commands translate into robot actions [58,23,30,37,40]. By utilizing demonstration data of how humans and robots perform each task, our approach learns the correspondence between the human and robot implicitly.…”

Section: Introductionmentioning

confidence: 99%

One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

Finn

Xie

et al. 2018

Robotics: Science and Systems XIV

219

192

View full text Add to dashboard Cite

Abstract-Humans and animals are capable of learning a new behavior by observing others perform the skill just once. We consider the problem of allowing a robot to do the same -learning from a video of a human, even when there is domain shift in the perspective, environment, and embodiment between the robot and the observed human. Prior approaches to this problem have hand-specified how human and robot actions correspond and often relied on explicit human pose detection systems. In this work, we present an approach for one-shot learning from a video of a human by using human and robot demonstration data from a variety of previous tasks to build up prior knowledge through meta-learning. Then, combining this prior knowledge and only a single video demonstration from a human, the robot can perform the task that the human demonstrated. We show experiments on both a PR2 arm and a Sawyer arm, demonstrating that after meta-learning, the robot can learn to place, push, and pick-andplace new objects using just one video of a human performing the manipulation.

show abstract

Deep Episodic Memory: Encoding, Recalling, and Predicting Episodic Experiences for Robot Action Execution

Cited by 32 publications

References 19 publications

Learning One-Shot Imitation From Humans Without Humans

Learning One-Shot Imitation From Humans Without Humans

Using the Engagement Profile to Design an Engaging Robotic Teaching Assistant for Students

One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

Contact Info

Product

Resources

About