An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents

Such, Felipe Petroski; Madhavan, Vashisht; Liu, Rosanne; Wang, Rui; Castro, Pablo Samuel; Li, Yulun; Zhi, Jiale; Schubert, Ludwig; Bellemare, Marc G.; Clune, Jeff; Lehman, Joel

doi:10.48550/arxiv.1812.07069

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Learned Representations We further investigate what type of world model can emerge from an evolutionary process that does not directly optimize for forward prediction or reconstruction loss. To gain insights into the learned representations we employ the t-SNE dimensionality reduction technique (Maaten and Hinton 2008), which has proven valuable for visualizing the inner workings of deep neural networks (Such et al 2018;Mnih et al 2015). We are particularly interested in the information contained in the compressed 32-dimensional vector of the VC and the information stored in the hidden states of the MDN-RNN (which are both fed into the controller that decides on the agent's action).…”

Section: Resultsmentioning

confidence: 99%

Deep Innovation Protection: Confronting the Credit Assignment Problem in Training Heterogeneous Neural Architectures

Risi

Stanley

2021

AAAI

View full text Add to dashboard Cite

Deep reinforcement learning approaches have shown impressive results in a variety of different domains, however, more complex heterogeneous architectures such as world models require the different neural components to be trained separately instead of end-to-end. While a simple genetic algorithm recently showed end-to-end training is possible, it failed to solve a more complex 3D task. This paper presents a method called Deep Innovation Protection (DIP) that addresses the credit assignment problem in training complex heterogenous neural network models end-to-end for such environments. The main idea behind the approach is to employ multiobjective optimization to temporally reduce the selection pressure on specific components in multi-component network, allowing other components to adapt. We investigate the emergent representations of these evolved networks, which learn to predict properties important for the survival of the agent, without the need for a specific forward-prediction loss.

show abstract

Section: Resultsmentioning

confidence: 99%

Deep Innovation Protection: Confronting the Credit Assignment Problem in Training Heterogeneous Neural Architectures

Risi

Stanley

2021

AAAI

View full text Add to dashboard Cite

show abstract

“…Manipulating the output image of deep generative models is achieved by editing the disentangled hidden features [5,57,66]. On the other hand, visuomotor control tasks can also acquire visual understanding to explain the behavior chosen by agent with visual input through saliency map [59,3,47,12,26,22].…”

Section: Related Workmentioning

confidence: 99%

Human-AI Shared Control via Policy Dissection

Li¹,

Peng²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Human-AI shared control allows human to interact and collaborate with AI to accomplish control tasks in complex environments. Previous Reinforcement Learning (RL) methods attempt the goal-conditioned design to achieve human-controllable policies at the cost of redesigning the reward function and training paradigm. Inspired by the neuroscience approach to investigate the motor cortex in primates, we develop a simple yet effective frequency-based approach called Policy Dissection to align the intermediate representation of the learned neural controller with the kinematic attributes of the agent behavior. Without modifying the neural controller or retraining the model, the proposed approach can convert a given RL-trained policy into a human-interactive policy. We evaluate the proposed approach on the RL tasks of autonomous driving and locomotion. The experiments show that human-AI shared control achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes. With human in the loop, the locomotion robots also exhibit versatile controllable motion skills even though they are only trained to move forward. Our results suggest the promising direction of implementing human-AI shared autonomy through interpreting the learned representation of the autonomous agents. Demo video and code will be made available at https://metadriverse.github.io/policydissect.Preprint. Under review.

show abstract

“…Diverse policy seeking methods The work of Such et al shows that different RL algorithms may converge to different policies for the same task [30]. On the contrary, we are interested in how to learn different policies through a single learning algorithm with the capability of avoiding local optimum.…”

Section: Related Workmentioning

confidence: 99%

Novel Policy Seeking with Constrained Optimization

Sun

Peng

Dai

et al. 2020

Preprint

View full text Add to dashboard Cite

In this work, we address the problem of seeking novel policies in reinforcement learning tasks. Instead of following the multi-objective framework commonly used in existing methods, we propose to rethink the problem under a novel perspective of constrained optimization. We at first introduce a new metric to evaluate the difference between policies, and then design two practical novel policy seeking methods following the new perspective, namely the Constrained Task Novel Bisector (CTNB), and the Interior Policy Differentiation (IPD), corresponding to the feasible direction method and the interior point method commonly known in the constrained optimization literature. Experimental comparisons on the Mu-JuCo control suite show our methods can achieve substantial improvements over previous novelty-seeking methods in terms of both novelty of policies and their performances in the primal task. 2

show abstract

An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents

Cited by 4 publications

References 0 publications

Deep Innovation Protection: Confronting the Credit Assignment Problem in Training Heterogeneous Neural Architectures

Deep Innovation Protection: Confronting the Credit Assignment Problem in Training Heterogeneous Neural Architectures

Human-AI Shared Control via Policy Dissection

Novel Policy Seeking with Constrained Optimization

Contact Info

Product

Resources

About