Improving Deep Reinforcement Learning in Minecraft with Action Advice

Abstract

Training deep reinforcement learning agents complex behaviors in 3D virtual environments requires significant computational resources. This is especially true in environments with high degrees of aliasing, where many states share nearly identical visual features. Minecraft is an exemplar of such an environment. We hypothesize that interactive machine learning (IML), wherein human teachers play a direct role in training through demonstrations, critique, or action advice, may alleviate agent susceptibility to aliasing. However, interactive machine learning is only practical when the number of human interactions is limited, requiring a balance between human teacher effort and agent performance. We conduct experiments with two reinforcement learning algorithms which enable human teachers to give action advice—Feedback Arbitration, and Newtonian Action Advice—under visual aliasing conditions. To assess potential cognitive load per advice type, we vary the accuracy and frequency of various human action advice techniques. The training efficiency, robustness against infrequent and inaccurate advisor input, and sensitivity to aliasing are examined.

Towards Goal-Oriented Agents for Evolving Problems Observed via Conversation

“…Solving navigational problems similar to the gridsworld we introduce is a natural problem upon which to apply reinforcement learning techniques. Examples include an RL agent designed to navigate a maze built in Minecraft [6]. The RL agent is augmented by a synthetic oracle, that gives one-hot-encoded directional advice to the agent.…”

“It’s Unwieldy and It Takes a Lot of Time” — Challenges and Opportunities for Creating Agents in Commercial Games

“…Imitation learning (Hussein et al 2017) is one way to intuitively exert designer control over what an agent learns by demonstrating desirable behaviours for the agent. Hybrid systems (Lee et al 2018;Neufeld, Mostaghim, and Brand 2018) or interactive machine learning approaches (Frazier and Riedl 2019) could also be re-purposed for this goal. Rapid iteration would also be aided by agents that could quickly and meaningfully be adapted after training instead of retraining from scratch.…”

Curriculum Goal-Conditioned Imitation for Offline Reinforcement Learning

“…Reinforcement learning (RL) has achieved remarkable success in a wide range of tasks, such as game playing (Mnih et al, 2013;Frazier & Riedl, 2019;Mao et al, 2022) and in robotics (Brunke et al, 2022;Nguyen & La, 2019). Goalconditioned RL (GCRL) (Liu et al, 2022b;Chane-Sane et al, 2021;Andrychowicz et al, 2017) allows us to learn a more general RL agent which can reach an arbitrary goal without retraining.…”