Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences

Abstract

Approximate Policy Iteration (API) algorithms alternate between (approximate) policy evaluation and (approximate) greedification. Many different approaches have been explored for approximate policy evaluation, but less is understood about approximate greedification and what choices guarantee policy improvement. In this work, we investigate approximate greedification when reducing the KL divergence between the parameterized policy and the Boltzmann distribution over action values. In particular, we investigate the difference between the forward and reverse KL divergences, with varying degrees of entropy regularization. We show that the reverse KL has stronger policy improvement guarantees, but that reducing the forward KL can result in a worse policy. We also demonstrate, however, that a large enough reduction of the forward KL can induce improvement under additional assumptions. Empirically, we show on simple continuous-action environments that the forward KL can induce more exploration, but at the cost of a more suboptimal policy. No significant differences were observed in the discrete-action setting or on a suite of benchmark problems. Throughout, we highlight that many policy gradient methods can be seen as an instance of API, with either the forward or reverse KL for the policy update, and discuss next steps for understanding and improving our policy optimization algorithms.

The Geometry of Robust Value Functions

“…Their results provide a geometric perspective to help understand the dynamics of different RL algorithms (Kumar et al, 2019;Chan et al, 2020;Harb et al, 2020;Chan et al, 2021), and also inspire new methods in representation learning in RL (Bellemare et al, 2019;Dabney et al, 2021).…”

Critic Sequential Monte Carlo

“…The discount factor γ is introduced to reduce variance and improve the convergence of soft-Q iteration [13,8]. For stability, we replace the bootstrap term Q ⊥(φ) with a φ-averaging target network Q ψ using the Polyak method [43].…”

“…Their derivation is based on the two-filter formula [10,36], but the method is equivalent to the one that would be obtained using the equations from Figure 1b with the heuristic factors set to be the value function h t = V φ (s t+1 ). This formulation can not accommodate putative action particles and uses a parametric policy to learn the value function instead of the soft Bellman update [5,13].…”

“…We propose a new SMC algorithm called critic sequential Monte Carlo (CriticSMC) that uses the soft Q-function [13,26] referred to as "critic" which bears similarity to classical Q-functions in the RL literature [63]. We employ these soft Q-functions in a novel way to encourage retention of high future-likelihood particle ancestries.…”

AdvNF: Reducing mode collapse in conditional normalising flows using adversarial learning

“…This causes q to choose any mode when p is multimodal, resulting in a concentration of probability density on that mode and ignoring the other high-density modes. Consequently, optimising RKL leads to a suboptimal solution, causing q to have limited support, leading to mode collapse [34][35][36]. In a sense, mode collapse and mode seeking are equivalent because mode-seeking leads to focusing on a few modes only, which causes mode collapse.…”