ViZDoom: DRQN with Prioritized Experience Replay, Double-Q Learning and Snapshot Ensembling

Abstract

ViZDoom is a robust, first-person shooter reinforcement learning environment, characterized by a significant degree of latent state information. In this paper, double-Q learning and prioritized experience replay methods are tested under a certain ViZDoom combat scenario using a competitive deep recurrent Q-network (DRQN) architecture. In addition, an ensembling technique known as snapshot ensembling is employed using a specific annealed learning rate to observe differences in ensembling efficacy under these two methods. Annealed learning rates are important in general to the training of deep neural network models, as they shake up the statusquo and counter a model's tending towards local optima. While both variants show performance exceeding those of built-in AI agents of the game, the known stabilizing effects of double-Q learning are illustrated, and priority experience replay is again validated in its usefulness by showing immediate results early on in agent development, with the caveat that value overestimation is accelerated in this case. In addition, some unique behaviors are observed to develop for priority experience replay (PER) and double-Q (DDQ) variants, and snapshot ensembling of both PER and DDQ proves a valuable method for improving performance of the ViZDoom Marine.

Learning Structures: Predictive Representations, Replay, and Generalization

“…Human and animal neural evidence supports a role for wake and sleep memory replay in generalization and community detection [59,60] , inference [52,61] especially of unseen relations among states, problem solving, and memory consolidation [62,63] . Deep reinforcement learning algorithms also benefit from prioritized replay in generalization, discovery, and adversarial self-learning [64][65][66] . Different replay algorithms differ with regards to how memories are stored and the model from which they generate the experience.…”

Optimizing for generalization in the decoding of internally generated activity in the hippocampus

“…We may find that these “replays” preferentially included the decision point at the middle of the track, rather than the ends of the track. We may be tempted to report this as a finding of interest, perhaps with an interpretation emphasizing prioritized replay as a mechanism useful for reinforcement learning (Schaul et al, ; Gershman and Daw, ). However, Figure should make it clear that, in the data set used here, such a bias is a straightforward consequence of the increase in cross‐validated decoding error at both ends of the track.…”

Ensemble Network Architecture for Deep Reinforcement Learning

“…The first one is that there are strong correlations among the incoming data, which may break the assumption of many popular stochastic gradient-based algorithms. Secondly, the minor changes in the function may result in a huge change in the policy, which makes the algorithm difficult to converge [7,9,14,15].…”

“…The convergence issue was mentioned in 2015 by Schaul et al [14]. The above -learning update rules can be directly implemented in a neural network.…”