Abstract-The recent advances in deep neural networks have led to effective vision-based reinforcement learning methods that have been employed to obtain human-level controllers in Atari 2600 games from pixel data. Atari 2600 games, however, do not resemble real-world tasks since they involve non-realistic 2D environments and the third-person perspective. Here, we propose a novel test-bed platform for reinforcement learning research from raw visual information which employs the firstperson perspective in a semi-realistic 3D world. The software, called ViZDoom, is based on the classical first-person shooter video game, Doom. It allows developing bots that play the game using the screen buffer. ViZDoom is lightweight, fast, and highly customizable via a convenient mechanism of user scenarios. In the experimental part, we test the environment by trying to learn bots for two scenarios: a basic move-and-shoot task and a more complex maze-navigation problem. Using convolutional deep neural networks with Q-learning and experience replay, for both scenarios, we were able to train competent bots, which exhibit human-like behaviors. The results confirm the utility of ViZDoom as an AI research platform and imply that visual reinforcement learning in 3D realistic first-person perspective environments is feasible.

Genetic programming (GP) is not a field noted for the rigor of its benchmarking. Some of its benchmark problems are popular purely through historical contingency, and they can be criticized as too easy or as providing misleading information concerning real-world performance, but they persist largely because of inertia and the lack of good alternatives. Even where the problems themselves are impeccable, comparisons between studies are made more difficult by the lack of standardization. We argue that the definition of standard benchmarks is an essential step in the maturation of the field. We make several contributions towards this goal. We motivate the development of a benchmark suite and define its goals; we survey existing practice; we enumerate many candidate benchmarks; we report progress on reference implementations; and we set out a concrete plan for gathering feedback from the GP community that would, if adopted, lead to a standard set of benchmarks.

We present the results of a community survey regarding genetic programming benchmark practices. Analysis shows broad consensus that improvement is needed in problem selection and experimental rigor. While views expressed in the survey dissuade us from proposing a large-scale benchmark suite, we find community support for creating a ''blacklist'' of problems which are in common use but have important flaws, and whose use should therefore be discouraged. We propose a set of possible replacement problems.

This paper presents the first two editions of Visual Doom AI Competition , held in 2016 and 2017. The challenge was to create bots that compete in a multi-player deathmatch in a first-person shooter (FPS) game, Doom. The bots had to make their decisions based solely on visual information, i.e., a raw screen buffer. To play well, the bots needed to understand their surroundings, navigate, explore, and handle the opponents at the same time. These aspects, together with the competitive multi-agent aspect of the game, make the competition a unique platform for evaluating the state of the art reinforcement learning algorithms. The paper discusses the rules, solutions, results, and statistics that give insight into the agents' behaviors. Bestperforming agents are described in more detail. The results of the competition lead to the conclusion that, although reinforcement learning can produce capable Doom bots, they still are not yet able to successfully compete against humans in this game. The paper also revisits the ViZDoom environment, which is a flexible, easy to use, and efficient 3D platform for research for vision-based reinforcement learning, based on a well-recognized first-person perspective game Doom.

Motor imagery has been argued to affect the acquisition of motor skills. The present study examined the specificity of motor imagery on the learning of a fine hand motor skill by employing a modified discrete sequence production task: the Go/NoGo DSP task. After an informative cue, a response sequence had either to be executed, imagined, or withheld. To establish learning effects, the experiment was divided into a practice phase and a test phase. In the latter phase, we compared mean response times and accuracy during the execution of unfamiliar sequences, familiar imagined sequences, and familiar executed sequences. The electroencephalogram was measured in the practice phase to compare activity between motor imagery, motor execution, and a control condition in which responses should be withheld. Event-related potentials (ERPs) and event-related lateralizations (ERLs) showed strong similarities above cortical motor areas on trials requiring motor imagery and motor execution, while a major difference was found with trials on which the response sequence should be withheld. Behavioral results from the test phase showed that response times and accuracy improved after physical and mental practice relative to unfamiliar sequences (so-called sequence-specific learning effects), although the effect of motor learning by motor imagery was smaller than the effect of physical practice. These findings confirm that motor imagery also resembles motor execution in the case of a fine hand motor skill.