Real-World Human-Robot Collaborative Reinforcement Learning

Abstract

The intuitive collaboration of humans and intelligent robots (embodied AI) in the real-world is an essential objective for many desirable applications of robotics. Whilst there is much research regarding explicit communication, we focus on how humans and robots interact implicitly, on motor adaptation level. We present a real-world setup of a humanrobot collaborative maze game, designed to be non-trivial and only solvable through collaboration, by limiting the actions to rotations of two orthogonal axes, and assigning each axes to one player. This results in neither the human nor the agent being able to solve the game on their own. We use a stateof-the-art reinforcement learning algorithm for the robotic agent, and achieve results within 30 minutes of real-world play, without any type of pre-training. We then use this system to perform systematic experiments on human/agent behaviour and adaptation when co-learning a policy for the collaborative game. We present results on how co-policy learning occurs over time between the human and the robotic agent resulting in each participant's agent serving as a representation of how they would play the game. This allows us to relate a person's success when playing with different agents than their own, by comparing the policy of the agent with that of their own agent.

Task dynamics define the contextual emergence of human corralling behaviors

“…Agents trained using deep reinforcement learning (DRL; i.e., the integration of reinforcement learning with deep neural networks), for example, have been successful in discovering adaptive behavior and strategies in individual [ 59 ] and group task contexts [ 60 , 61 ]. Within the context of working with humans in collaborative tasks, such agents can develop control policies that are either user-specific [ 62 ] or generalize to a distribution of human strategies during training [ 63 ]. By giving meaning to actions with the use of reward functions [ 64 ], black-box self-supervised approaches have the ability to provide a “direct fit” [ 65 ] between an agent and task-relevant states–assuming there is sufficient sampling of the task environment.…”

A Comparison of Dynamical Perceptual-Motor Primitives and Deep Reinforcement Learning for Human-Artificial Agent Training Systems

“…Indeed, a substantial limitation or impediment to the development of DPMP models is that they require researchers and modelers to have a thorough a-priori understanding of the task dynamics that ensure task success, which often involves a significant amount of experimental research and data-driven optimization. In contrast, DRL methods do not require any prior knowledge of the dynamics of the task environment or the agent’s actions, and thus show great promise in developing flexible strategies and interaction couplings that are either user-specific (Shafti et al, 2020) or can generalize to a diversity of users (Carroll et al, 2019). However, it is also the case that DRL methods are often notoriously slow and computationally expensive, thus restricting their use in cases of applications with sparse rewards or which lack access to datasets of desired behavior.…”

“…Atari 2600 games (Bellemare et al, 2012), DOTA (Berner et al, 2019), Starcraft II (Vinyals et al, 2019), with such video games serving as the benchmark for DRL testing and development. More recently, research has also demonstrated how DRL agents can extend beyond simulated environments to achieve successful multiagent performance in physical system tasks (Shafti et al, 2020; Morgan et al, 2021).…”

Becoming Team Members: Identifying Interaction Patterns of Mutual Adaptation for Human-Robot Co-Learning

“…( Nikolaidis et al, 2017b ; Mohammad and Nishida., 2008 ; Nikolaidis et al, 2017a )]. The studies that use “co-learning” tend to take a more symmetrical approach by looking at agent or robot learning as well as human learning, and pay more attention to the learning process and changing strategies of the human as well, often looking at many repetitions of a task ( Ramakrishnan, Zhang, and Shah 2017 ; C.-S. Lee et al, 2020 ; C. Lee et al, 2018 ; Shafti et al, 2020 ). Studies on co-evolution, on the other hand, monitor a long-term real-world application in which behavior of the human as well as the robot subtly changes over time ( Döppner, Derckx, and Schoder 2019 ).…”