Collaborative Deep Reinforcement Learning

Abstract

Besides independent learning, human learning process is highly improved by summarizing what has been learned, communicating it with peers, and subsequently fusing knowledge from di erent sources to assist the current learning goal. is collaborative learning procedure ensures that the knowledge is shared, continuously re ned, and concluded from di erent perspectives to construct a more profound understanding. e idea of knowledge transfer has led to many advances in machine learning and data mining, but signi cant challenges remain, especially when it comes to reinforcement learning, heterogeneous model structures, and di erent learning tasks. Motivated by human collaborative learning, in this paper we propose a collaborative deep reinforcement learning (CDRL) framework that performs adaptive knowledge transfer among heterogeneous learning agents. Speci cally, the proposed CDRL conducts a novel deep knowledge distillation method to address the heterogeneity among di erent learning tasks with a deep alignment network. Furthermore, we present an e cient collaborative Asynchronous Advantage Actor-Critic (cA3C) algorithm to incorporate deep knowledge distillation into the online training of agents, and demonstrate the e ectiveness of the CDRL framework using extensive empirical evaluation on OpenAI gym.

Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networks

“…Hence, policy training for the CDRL agents must be optimized jointly with the source agent selection and wireless resource allocation. Given these challenges, the problem in (4a)-(4d) cannot be solved via existing CDRL methods such as knowledge distillation [11]- [14] to transfer the source agents' knowledge to the target agent. Moreover, existing CDRL methods cannot be directly applied here as they do not account for the wireless and real-time constraints of knowledge sharing among agents.…”

“…where f n (θ n ) represents the loss function of the local model for agent n ∈ N . Considering policy gradient (PG) DRL algorithms [11], the loss function is defined as…”

“…Moreover, enabling an effective CDRL among heterogeneous agents (as expected in IoT applications) is very challenging, due to dissimilarities of agents (e.g., different action spaces), environments, and diversity of DRL tasks. In the CDRL context, the heterogeneity of environments, modeled as Markov decision processes (MDPs), as well as agents and their tasks can be expressed in two main forms: 1) distinct DRL tasks that are conceptually similar (i.e., semantically related tasks) or completely dissimilar [8], [9] and 2) distinct environments represented by different MDPs [10], [11]. Most existing works, such as in [5]- [7], study CDRL among homogeneous agents, i.e., agents with the same action space.…”

“…Most existing works, such as in [5]- [7], study CDRL among homogeneous agents, i.e., agents with the same action space. Meanwhile, the works in [8]- [11] consider more realistic scenarios by adopting knowledge transfer for CDRL among heterogeneous agents. While interesting, most of the prior art in [8]- [11] assumes that an expert agent is always available and DRL tasks are semantically similarstrong assumptions that are often inapplicable to practical IoT scenarios.…”

Dual Policy Distillation

“…Our framework also learns from imperfect demonstrations, but treats actions performed by a peer policy as demonstrations. Collaborative learning is also studied in [Lin et al, 2017], however, with expensive pre-trained teachers. Meta-learning methods also make use of a teacher model to improve the sample efficiency [Xu et al, 2018b;Xu et al, 2018a;Zha et al, 2019b].…”