The AI Driving Olympics at NeurIPS 2018

Abstract

Despite recent breakthroughs, the ability of deep learning and reinforcement learning to outperform traditional approaches to control physically embodied robotic agents remains largely unproven. To help bridge this gap, we created the "AI Driving Olympics" (AI-DO), a competition with the objective of evaluating the state of the art in machine learning and artificial intelligence for mobile robotics. Based on the simple and well specified autonomous driving and navigation environment called "Duckietown," AI-DO includes a series of tasks of increasing complexity -from simple lane-following to fleet management. For each task, we provide tools for competitors to use in the form of simulators, logs, code templates, baseline implementations and low-cost access to robotic hardware. We evaluate submissions in simulation online, on standardized hardware environments, and finally at the competition event. The first AI-DO, AI-DO 1, occurred at the Neural Information Processing Systems (NeurIPS) conference in December 2018. The results of AI-DO 1 highlight the need for better benchmarks, which are lacking in robotics, as well as improved mechanisms to bridge the gap between simulation and reality.

Vision-based reinforcement learning for lane-tracking control

“…In this study, a neural-network-based controller was trained that takes images from a forward-looking monocular camera and produces control signals to drive a vehicle in the right-hand lane of a two-way road. The vehicle to be controlled was a small differential-wheeled mobile robot, a Duckiebot, which is part of the Duckietown ecosystem [11], a simple and accessible platform for research and education on mobile robotics and autonomous vehicles. The primary objective was to travel as far as possible within a given time without leaving the road.…”

“…This paper is an extended version of the authors' original contribution [10]. It includes the results of the 5 th AI Driving Olympics [11] and aims to improve the description of the methods. In both works, vision-based end-to-end reinforcement learning relating to vehicle control problems is studied and a solution is proposed that performs lane following in the real world, using continuous actions, without any real data provided by an expert (as in [3]).…”

Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents

“…The DUCKIENet is built using the Duckietown platform, which was originally developed in 2016 and has since been used for education [27], research [28] and outreach [29]. 2) The Duckietown software architecture: We implement the Duckietown base software as ROS nodes and use the ROS topic messaging system for inter-process communication.…”

“…It is possible to set thresholds on the various scores that need to be reached for progression. For competitions such as the AI Driving Olympics [28], the superusers can set up conditions such as "progress to the next stage only if the submission does better than a baseline".…”

“…One of the key applications of our infrastructure is to host competitions. The AI Driving Olympics (AI-DO) [28] is a bi-annual competition that leverages the DUCKIENet to compare the ability of various agents, ranging from traditional estimation-planning-control architectures to end-to-end learning-based architectures, to perform autonomous driving tasks of increasing complexity. By fixing the environment and the hardware, agents and tasks can be modified to benchmark performance across different challenges.…”