On the Role of Weight Sharing During Deep Option Learning

Abstract

The options framework is a popular approach for building temporally extended actions in reinforcement learning. In particular, the option-critic architecture provides general purpose policy gradient theorems for learning actions from scratch that are extended in time. However, past work makes the key assumption that each of the components of option-critic has independent parameters. In this work we note that while this key assumption of the policy gradient theorems of option-critic holds in the tabular case, it is always violated in practice for the deep function approximation setting. We thus reconsider this assumption and consider more general extensions of option-critic and hierarchical option-critic training that optimize for the full architecture with each update. It turns out that not assuming parameter independence challenges a belief in prior work that training the policy over options can be disentangled from the dynamics of the underlying options. In fact, learning can be sped up by focusing the policy over options on states where options are actually likely to terminate. We put our new algorithms to the test in application to sample efficient learning of Atari games, and demonstrate significantly improved stability and faster convergence when learning long options. 1

Hierarchical Average Reward Policy Gradient Algorithms (Student Abstract)

“…the agent reaches the blue zone, it obtains a reward of +20 as opposed to a reward of +10 at the red-green junction. In Figure 1, we plot the rewards obtained per cycle for both the AR-RL agent and a DR-RL agent, and show that the hierarchical AR policy gradient performs better than its DR counterpart proposed by Riemer et al (2020). Finally, we illustrate the asymptotic convergence of the actor and critic parameters in Figure 2.…”

“…Finally, we look at the susceptibility of our framework to traps, and compare it to the DR setting proposed by Riemer et al (2020). Figure 3(b) depicts a grid world environment characterized by sparse rewards.…”

“…First, we illustrate how to extend the framework proposed by Riemer et al (2020) for the AR criterion. Apart from addressing the AR criterion, our framework also presents a simplified and intuitive approach to dealing with hierarchical option-critic algorithms (Riemer, Liu, and Tesauro 2018) by introducing the concept of o 0 and o N .…”

“…We can now follow a similar procedure as the one explored (Riemer et al 2020), and take the derivative of Q Ω (s, o 0:N −1 ) with respect to θ. By incorporating the notion of o 0 into our equations, we were able to significantly reduce the complexity of the following equations.…”

“…However, employing a discount factor to bound the cumulative rewards can inadvertently lead to incorrect credit assignment. We addresses this issue by extending the framework proposed by Riemer et al (2020) for the average reward (AR) criterion.…”