Revisiting the Softmax Bellman Operator: New Benefits and New Perspective

Abstract

The impact of softmax on the value function itself in reinforcement learning (RL) is often viewed as problematic because it leads to sub-optimal value (or Q) functions and interferes with the contraction properties of the Bellman operator. Surprisingly, despite these concerns, and independent of its effect on exploration, the softmax Bellman operator when combined with Deep Q-learning, leads to Q-functions with superior policies in practice, even outperforming its double Q-learning counterpart. To better understand how and why this occurs, we revisit theoretical properties of the softmax Bellman operator, and prove that (i) it converges to the standard Bellman operator exponentially fast in the inverse temperature parameter, and (ii) the distance of its Q function from the optimal one can be bounded. These alone do not explain its superior performance, so we also show that the softmax operator can reduce the overestimation error, which may give some insight into why a sub-optimal operator leads to better performance in the presence of value function approximation. A comparison among different Bellman operators is then presented, showing the trade-offs when selecting them.

Reinforcement Learning with Dynamic Boltzmann Softmax Updates

“…Therefore, the DBS operator rectifies the convergence issue of the Boltzmann softmax operator with fixed parameters. Note that we also achieve a tighter error bound for the fixed-parameter softmax operator in general cases compared with Song et al (2018). In addition, we show that the DBS operator achieves good convergence rate.…”

“…In this section, we compare the error bound in Corollary 2 with that in Song et al (2018), which studies the error bound of the softmax operator with a fixed parameter β.…”

“…The Boltzmann softmax operator is a natural value estimator Sutton & Barto (1998); Azar et al (2012); Cesa-Bianchi et al (2017) based on the Boltzmann softmax distribution, which is a natural scheme to address the exploration-exploitation dilemma and has been widely used in reinforcement learning Sutton & Barto (1998); Azar et al (2012); Cesa-Bianchi et al (2017). In addition, the Boltzmann softmax operator also provides benefits for reducing overestimation and gradient noise in deep Q-networks Song et al (2018). However, despite the advantages, it is challenging to apply the Boltzmann softmax operator in value function estimation.…”

“…It is crucial to note the DBS operator is the only one that meets all desired properties proposed in Song et al (2018) up to now, as it ensures Bellman optimality, enables overestimation reduction, directly represents a policy, can be applicable to double Q-learning Hasselt (2010), and requires no tuning.…”

Ensemble Consensus-based Representation Deep Reinforcement Learning for Hybrid FSO/RF Communication Systems

“…These layers are implemented using ReLu activation functions with the linear output layer. For both the DQNEnsemble-FSO/RF and DQN-FSO/RF agents, actions are selected using the Boltzman policy [41]. Our evaluations consider the use of Adam optimizer to minimize the loss function given in Equation 17.…”