Accelerating Interactive Reinforcement Learning by Human Advice for an Assembly Task by a Cobot

Abstract

The assembly industry is shifting more towards customizable products, or requiring assembly of small batches. This requires a lot of reprogramming, which is expensive because a specialized engineer is required. It would be an improvement if untrained workers could help a cobot to learn an assembly sequence by giving advice. Learning an assembly sequence is a hard task for a cobot, because the solution space increases drastically when the complexity of the task increases. This work introduces a novel method where human knowledge is used to reduce this solution space, and as a result increases the learning speed. The method proposed is the IRL-PBRS method, which uses Interactive Reinforcement Learning (IRL) to learn from human advice in an interactive way, and uses Potential Based Reward Shaping (PBRS), in a simulated environment, to focus learning on a smaller part of the solution space. The method was compared in simulation to two other feedback strategies. The results show that IRL-PBRS converges more quickly to a valid assembly sequence policy and does this with the fewest human interactions. Finally, a use case is presented where participants were asked to program an assembly task. Here, the results show that IRL-PBRS learns quickly enough to keep up with advice given by a user, and is able to adapt online to a changing knowledge base.

A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications

“…Applications of human-in-the-loop reinforcement learning to deal with real-world problems have been investigated in few papers. For example, in Shah et al [ 78 ], a human-in-the-loop reinforcement learning approach was presented for task-oriented dialogue management and in Winter et al [ 79 ], to program an assembly task, human-in-the-loop reinforcement learning approach was used.…”

Transparent Interaction Based Learning for Human-Robot Collaboration

“…In our previous work (De Winter et al, 2019), we applied an Interactive Reinforcement Learning (IRL) model to enable a cobot to assemble the Cranfield benchmark (Collins et al, 1985). The Cranfield benchmark comprises of 9 objects: base plate, top plate, square peg (×2), round peg (×2), pendulum, separator, and shaft (Figure 1).…”

“…The challenge in (De Winter et al, 2019) was to teach the cobot the correct order of the objects to assemble the Cranfield benchmark correctly, which IRL was able to achieve. The applied IRL model generates a Q-table, which shows the appropriateness of selecting each object in different states.…”

“…One solution to both reduce the cost of teaching cobots new tasks and increase their flexibility (being able to perform different but similar tasks) is enabling their human partners to teach them new tasks, such that a cobot learns new high-level tasks from a nonprogrammer, e.g., an operator in a workshop, who knows the necessary steps to perform the task but does not know how to program the cobot (Billard et al, 2016;Huang and Cakmak, 2017;Stenmark et al, 2017). This way, the costs of cobot maintenance are reduced by transferring behaviors instead of reprogramming them (De Winter et al, 2019). However, this approach requires interactions between cobots and humans, where cobots can clarify, explain, and justify their behaviors for the human partners, and the human partners can transfer their knowledge through feedback to the cobots to help them perform their tasks.…”

A framework for flexibly guiding learning agents

“…The use of prior knowledge in the form of advice, feedback, and demonstrations has been proven to significantly increase the sample efficiency of the RL algorithms in single-agent settings e.g. [27,17,16,25,21,6]. But little has been done in the area in multi-agent settings e.g.…”