Leveraging human knowledge in tabular reinforcement learning: a study of human subjects

Abstract

Reinforcement Learning (RL) can be extremely effective in solving complex, real-world problems. However, injecting human knowledge into an RL agent may require extensive effort and expertise on the human designer's part. To date, human factors are generally not considered in the development and evaluation of possible RL approaches. In this article, we set out to investigate how different methods for injecting human knowledge are applied, in practice, by human designers of varying levels of knowledge and skill. We perform the first empirical evaluation of several methods, including a newly proposed method named SASS which is based on the notion of similarities in the agent's state-action space. Through this human study, consisting of 51 human participants, we shed new light on the human factors that play a key role in RL. We find that the classical reward shaping technique seems to be the most natural method for most designers, both expert and non-expert, to speed up RL. However, we further find that our proposed method SASS can be effectively and efficiently combined with reward shaping, and provides a beneficial alternative to using only a single speedup method with minimal human designer effort overhead.

Q-Learning Algorithms: A Comprehensive Classification and Applications

“…In addition, it can usefully interact in real environments, reducing human supervision costs and being applied to state-of-the-art RL systems [89]. Based on human psychology, both non specialists and experts are effective, and research on putting human knowledge into Q-learning agents for speed improvement is underway [90].…”

A Survey on Transfer Learning for Multiagent Reinforcement Learning Systems

“…They show that natural language advice can present a better performance than regular learning while facilitating the inclusion of laypeople in the agent training process. Rosenfeld et al (2017) propose a method for reusing knowledge from humans with technical knowledge on programming and general AI, but without expertise on RL. The human defines a metric that estimates similarities between state-action pairs in the problem space.…”

“…Papers Gridworld (Tan, 1993), (Maclin et al, 1996), (Wiewiora et al, 2003), (Price & Boutilier, 2003), (Madden & Howley, 2004), (Sherstov & Stone, 2005), (Kolter et al, 2008), (Vrancx et al, 2011), (Boutsioukis et al, 2011), (Koga et al, 2013), (Kono et al, 2014), (Devlin et al, 2014), (Koga et al, 2015), (Hu et al, 2015a), (Freire & Costa, 2015), (Zhou et al, 2016), (Zhan et al, 2016), (Subramanian et al, 2016), (Silva & Costa, 2017b), (Rosenfeld et al, 2017), (Svetlik et al, 2017), (Narvekar et al, 2017), (Hernandez-Leal & Kaisers, 2017), (Chalmers et al, 2017), (Mandel et al, 2017).…”

A Novel Method for Improving the Training Efficiency of Deep Multi-Agent Reinforcement Learning

“…In the domain of MARL, there is much work on how to improve the efficiency of agent training. By combining transfer learning(TL) with reinforcement learning, knowledge reuse can be realized to accelerate agent learning process, such as inter-agent learning through the teacher-student model [31] and introducing human knowledge [32] in the training process. Some work improves the training efficiency of agents by sharing parameters or gradients [25], [33] among agents.…”