A Mutual Information-Based Assessment of Reverse Engineering on Rewards of Reinforcement Learning

Abstract

Rewards are critical hyperparameters in reinforcement learning (RL), since in most cases different reward values will lead to greatly different performance. Due to their commercial value, RL rewards become the target of reverse engineering by the inverse reinforcement learning (IRL) algorithm family. Existing efforts typically utilize two metrics to measure the IRL performance: the expected value difference and the mean reward loss, which we call them EVD and MRL respectively. Unfortunately, in some cases, EVD and MRL can give completely opposite results, due to MRL focusing on whole state-space rewards while EVD only considering partly sampled rewards. Such situation naturally rises to one fundamental question: whether current metrics and assessment are sufficient and accurate for more general use. Thus, in this paper, based on the metric called normalized mutual information of reward clusters (C-NMI) we propose a novel IRL assessment; we aim to fill this research gap by considering a middle-granularity state space between the entire state space and the specific sampling space. We utilize the agglomerative nesting algorithm (AGNES) to control dynamical C-NMI computing via a 4-order tensor model with injected manipulated trajectories. With such a model, we can uniformly capture different-dimension values of MRL, EVD, and C-NMI, and perform more comprehensive and accurate assessment and analyses. Extensive experiments on several mainstream IRLs are experimented in Object World, hence revealing that the assessing accuracy of our method increases 110.13% and 116.59% respectively when compared with the EVD and MRL. Meanwhile, C-NMI is more robust than EVD and MRL under different demonstrations.Impact Statement-In this work, we pay attention to the inconformity problem of MRL-and EVD-based IRL assessment. There are two main challenges for us to address: (1) how to design a novel metric by combining the advantages of both MRL and EVD, and (2) how to construct a comprehensive assessment method for accurate comparison and analysis. To address such challenges, we craft a novel assessment of IRL based on the metric called normalized mutual information of reward clusters (C-NMI). Hence we attempt to fill the existing research gap by considering a middle-granularity state space between the entire state space and the certain sampled space. We list all of the notation and parameters used in the rest of this paper in Table I.