The MineRL BASALT Competition on Learning from Human Feedback

Abstract

The last decade has seen a significant increase of interest in deep learning research, with many public successes that have demonstrated its potential. As such, these systems are now being incorporated into commercial products. With this comes an additional challenge: how can we build AI systems that solve tasks where there is not a crisp, well-defined specification? While multiple solutions have been proposed, in this competition we focus on one in particular: learning from human feedback. Rather than training AI systems using a predefined reward function or using a labeled dataset with a predefined set of categories, we instead train the AI system using a learning signal derived from some form of human feedback, which can evolve over time as the understanding of the task changes, or as the capabilities of the AI system improve.The MineRL BASALT competition aims to spur forward research on this important class of techniques. We design a suite of four tasks in Minecraft for which we expect it will be hard to write down hardcoded reward functions. These tasks are defined by a paragraph of natural language: for example, "create a waterfall and take a scenic picture of it", with additional clarifying details. Participants must train a separate agent for each task, using any method they want. Agents are then evaluated by humans who have read the task description. To help participants get started, we provide a dataset of human demonstrations on each of the four tasks, as well as an imitation learning baseline that leverages these demonstrations.

TEACh: Task-Driven Embodied Agents That Chat

“…More complex tasks involve partially observable worlds (Kim et al 2020) and object state changes (Misra et al 2018;Puig et al 2018;Shridhar et al 2020). Some works use a planner to generate ideal demonstrations that are then labeled, while others first gather instructions and gather human demonstrations (Misra et al 2018;Shah et al 2021;Abramson et al 2020). In TEACh, human instructions and demonstrations are gathered simultaneously.…”

Correct Me If I'm Wrong: Using Non-Experts to Repair Reinforcement Learning Policies

“…Human feedback can enable agents to solve sequential decision-making tasks for which the rewards are not easily defined [4]. Prior work used NE input to author [5], [6] or interactively shape robot behaviors [7], accelerate the learning, guide exploration, and prevent undesired actions [8]- [11].…”

MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learned

“…The year 2021 saw a number of Minecraft competitions that were organized alongside this MineRL Diamond competition. A sister competition, MineRL BASALT (Shah et al, 2021), made use of the same MineRL library to express desired agent goals with human demonstrations instead of the rewards, and then used human evaluators to rate the agents on how well they completed the tasks. The IGLU competition (Kiseleva et al, 2021) used the building mechanics of Minecraft for two challenges for the participants: build an agent that provides descriptions on what to build, and then build an agent which builds a structure based on the instructions.…”