Free-Lunch Saliency via Attention in Atari Agents

Abstract

We propose a new approach to visualize saliency maps for deep neural network models and apply it to deep reinforcement learning agents trained on Atari environments. Our method adds an attention module that we call FLS (Free Lunch Saliency) to the feature extractor from an established baseline [24]. This addition results in a trainable model that can produce saliency maps, i.e., visualizations of the importance of different parts of the input for the agent's current decision making. We show experimentally that a network with an FLS module exhibits performance similar to the baseline (i.e., it is "free", with no performance cost) and can be used as a drop-in replacement for reinforcement learning agents. We also design another feature extractor that scores slightly lower but provides higher-fidelity visualizations. In addition to attained scores, we report saliency metrics evaluated on the Atari-HEAD dataset of human gameplay.

Reinforcement Learning Interpretation Methods: A Survey

“…For example, pixels with higher value in the output image may indicate higher importance for their corresponding locations in the input image. As a result, changing the values of these pixels in the input image should affect the output more significantly [2]. The generated image can be either 1) local to explain the model decision for a specific image, or 2) global to generate the important pixels for a chosen class across the model as a whole.…”

“…Generally, there are different methods to generate saliency as reviewed by Nikulin at el. [2]. They investigated these methods and categorized them into post-hoc and built-in saliency maps.…”

“…Post-hoc methods were used broadly in different formats to explain ML [2]. For instance: 1) Guided BackProp Springenberg et al [45] proposed this method to test the importance of pooling layers in ConvNets.…”

“…Instead of using another model to generate saliency maps, in this interpretation method the original black-box model is altered to produce the maps in addition to the predictions. Altering the model to generate saliency maps beside its main objective most probably results in having a lower performance, which makes it harder to keep both interpretability and accuracy [2].…”

“…Nonetheless, RL interpretability lacks the same variety and depth of survey papers, when compared to ML interpretability, despite the fact that some researchers made a good effort in reviewing and categorizing the work related to their proposed approaches. For example, Nikulin et al [2] classified saliency maps applied on RL systems into ad-hoc and built in.…”