A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning

Abstract

We develop a mathematical framework for solving multi-task reinforcement learning problems based on a type of decentralized policy gradient method. The goal in multi-task reinforcement learning is to learn a common policy that operates effectively in different environments; these environments have similar (or overlapping) state and action spaces, but have different rewards and dynamics. Agents immersed in each of these environments communicate with other agents by sharing their models (i.e. their policy parameterizations) but not their state/reward paths. Our analysis provides a convergence rate for a consensus-based distributed, entropy-regularized policy gradient method for finding such a policy. We demonstrate the effectiveness of the proposed method using a series of numerical experiments. These experiments range from small-scale "Grid World" problems that readily demonstrate the trade-offs involved in multi-task learning to large-scale problems, where common policies are learned to play multiple Atari games or to navigate an airborne drone in multiple (simulated) environments.Preprint. Under review.

Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees

“…Additionally, the shared parameters are trained using more data (i.e., data drawn from all agents) compared with the personalized parameters, so the variance in the training process can be significantly reduced, potentially resulting in better training performance. Such an intuition has been verified empirically in multi-task RL systems [33,34], where sharing policies among different learners results in more stable convergence.…”

“…the homogeneous setting), the agent's policy should also be closely related to each other. Such an intuition has been verified empirically in MARL systems [31,32], as well as in multi-task RL systems [33,34], where sharing policies among different learners results in more stable convergence and/or better feature extraction. However, it is not clear how to design and analyze more sophisticated collaboration schemes which enable the agents to (partially) share their local policies to help them leverage each other's past experience and build better behavior strategies.…”

“…Unfortunately, in the cooperative MARL setting, existing algorithms either lack any finite-sample guarantees, or they have high sampling cost. In particular, the PG based methods [30,33,13] do provide convergence guarantees to show the relationship between the total number of iterations and the convergence error. However, at each iteration steps, they need to simulate global trajectories for each agent to evaluate their policies, which inevitably requires a large number of locally generated samples and are thus still incur very high sampling cost [35,36].…”

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

“…While our DSA framework is fairly general, a key limitation is that the scaling matrix (i.e., A) in each component function h i needs to be the same. It would be interesting to see if our approach can be extended to cover the general case Zeng et al [2020a] where the scaling matrices also depend on i. Another intriguing future direction is the setting with dynamic communication protocols, wherein the gossip matrix also evolves with time Doan et al [2019Doan et al [ , 2021.…”

“…The first is the CLT shown in Morral et al [2017] for the average of estimates obtained at different nodes in a generic DSA scheme. The other is the convergence in mean result obtained in Zeng et al [2020a] for a distributed policy gradient method.…”