Action Branching Architectures for Deep Reinforcement Learning

Abstract

Discrete-action algorithms have been central to numerous recent successes of deep reinforcement learning. However, applying these algorithms to high-dimensional action tasks requires tackling the combinatorial increase of the number of possible actions with the number of action dimensions. This problem is further exacerbated for continuous-action tasks that require fine control of actions via discretization. In this paper, we propose a novel neural architecture featuring a shared decision module followed by several network branches, one for each action dimension. This approach achieves a linear increase of the number of network outputs with the number of degrees of freedom by allowing a level of independence for each individual action dimension. To illustrate the approach, we present a novel agent, called Branching Dueling Q-Network (BDQ), as a branching variant of the Dueling Double Deep Q-Network (Dueling DDQN). We evaluate the performance of our agent on a set of challenging continuous control tasks. The empirical results show that the proposed agent scales gracefully to environments with increasing action dimensionality and indicate the significance of the shared decision module in coordination of the distributed action branches. Furthermore, we show that the proposed agent performs competitively against a state-of-the-art continuous control algorithm, Deep Deterministic Policy Gradient (DDPG).

Weighted-QMIX-based Optimization for Maintenance Decision-making of Multi-component Systems

“…To tackle this problem, the BDQ algorithm in (Tavakoli et al, 2018) provides a special network structure, namely, branching dueling deep Q-network (or branching network for short), that allows the number of outputs of deep Q-networks to linearly increases with the number of components as illustrated in figure 2.…”

“…To address these issues, in this paper, we customize WQMIX to effectively optimize maintenance decisions of large-scale multi-component systems for the fully observable setting. In particular, separate agent networks are replaced by a single branching dueling network (branching network) (Tavakoli, Pardo, & Kormushev, 2018) to take advantage of the fully observable setting. The branching structure allows achieving a linear increase in the size of deep Q-networks's output layer to avoid the cure of dimentionality.…”

“…Firstly, we customize WQMIX algorithm specifically for the fully observable setting . Secondly, we conduct a comparison study to benchmark the performance of the customized algorithm, the branching dueling deep Q-learning (Tavakoli et al, 2018) and a threshold-based policy when they are used to optimize maintenance actions of large-scale systems.…”

Artificial-Intelligence-Based Maintenance Scheduling for Complex Systems with Multiple Dependencies

“…In order to take the fourth expected contribution into account, we customize a MADRL algorithm, namely WQMIX (Rashid, Farquhar, Peng, & Whiteson, 2020), in the case where system states can be fully observed to obtain cost-effective policies. Indeed, the algorithm takes advantage of the branching dueling network architecture (Tavakoli, Pardo, & Kormushev, 2018) to allow achieving linear increase in the size of the output layer of deep Q-networks when the number of system components grows and the monotonic decomposition scheme for joint action-value functions (Rashid et al, 2020) to enable maintenance decision-making consistency at component and system level.…”

Commission Fee is not Enough: A Hierarchical Reinforced Framework for Portfolio Management

“…The low-level problem is considered as a discrete control problem with two action dimensions: price and quantity. We utilize the Branching Dueling Q-Network (Tavakoli, Pardo, and Kormushev 2018). Formally, we have two action dimensions with |p l | = n p discrete relative price levels and |q l | = n q discrete quantity proportions.…”