An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents

Abstract

Much human and computational effort has aimed to improve how deep reinforcement learning (DRL) algorithms perform on benchmarks such as the Atari Learning Environment. Comparatively less effort has focused on understanding what has been learned by such methods, and investigating and comparing the representations learned by different families of DRL algorithms. Sources of friction include the onerous computational requirements, and general logistical and architectural complications for running DRL algorithms at scale. We lessen this friction, by (1) training several algorithms at scale and releasing trained models, (2) integrating with a previous DRL model release, and (3) releasing code that makes it easy for anyone to load, visualize, and analyze such models. This paper introduces the Atari Zoo framework, which contains models trained across benchmark Atari games, in an easy-to-use format, as well as code that implements common modes of analysis and connects such models to a popular neural network visualization library. Further, to demonstrate the potential of this dataset and software package, we show initial quantitative and qualitative comparisons between the performance and representations of several DRL algorithms, highlighting interesting and previously unknown distinctions between them.

An opponent striatal circuit for distributional reinforcement learning

“…In RL settings, in contrast, uncertainty generally arises from a combination of state ambiguity, insufficient exploration, and intrinsic stochasticity 123 , all of which complicate the problem of learning from limited experience. Distributional RL excels in partitioning out this intrinsic uncertainty from other sources, potentially allowing for improvements in state representation 77,78 , exploration [79][80][81][82] , value estimation 124 , model-based learning 125 , off-policy learning 126 , and risk sensitivity [127][128][129][130] .…”

“…In addition to supporting our mechanistic REDRL model, the selective disruption of variance coding by 6-OHDA gives us an experimental tool with which to probe the function of distributional RL in the brain. When paired with deep neural networks, distributional RL is thought to boost system performance mainly by improving state representations 1,4,78 . Due to multiplexing of odor-specific representations alongside distribution information within the striatum (Extended Data Fig.…”

“…Given this, it is perhaps surprising that, with a few exceptions 66 , conventional RL models of the basal ganglia ignore the role of risk, and most theoretical investigations of uncertainty focus on sensory noise rather than intrinsic, irreducible environmental stochasticity [67][68][69][70] .Borrowing from tremendous successes and popularity in machine learning [71][72][73] , it was recently proposed 4 that the residual heterogeneity within RPE-coding dopamine neurons 74,75 resembles the predictions of a particular algorithm known as Expectile Distributional RL (EDRL) 76 . This algorithm not only unifies the learning of value and risk (and potentially higher-order moments of reward distributions) within the same biologically-plausible framework but also provides novel computational advantageseven with risk-neutral settingsrelated to representation learning in deep neural networks 77,78 and, potentially, directed exploration [79][80][81][82] . However,…”