Multi-agent Reinforcement Learning for Resource Allocation in IoT networks with Edge Computing

Abstract

To support popular Internet of Things (IoT) applications such as virtual reality, mobile games and wearable devices, edge computing provides a front-end distributed computing archetype of centralized cloud computing with low latency. However, it's challenging for end users to offload computation due to their massive requirements on spectrum and computation resources and frequent requests on Radio Access Technology (RAT). In this paper, we investigate computation offloading mechanism with resource allocation in IoT edge computing networks by formulating it as a stochastic game. Here, each end user is a learning agent observing its local environment to learn optimal decisions on either local computing or edge computing with the goal of minimizing long term system cost by choosing its transmit power level, RAT and sub-channel without knowing any information of the other end users. Therefore, a multiagent reinforcement learning framework is developed to solve the stochastic game with a proposed independent learners based multi-agent Q-learning (IL-based MA-Q) algorithm. Simulations demonstrate that the proposed IL-based MA-Q algorithm is feasible to solve the formulated problem and is more energy efficient without extra cost on channel estimation at the centralized gateway compared to the other two benchmark algorithms.

Cooperative Task Offloading and Block Mining in Blockchain-based Edge Computing with Multi-agent Deep Reinforcement Learning

“…Another study in [18] also suggested an MA-DRL approach for joint data offloading and resource allocation in multiple independent edge clouds. Furthermore, a multiagent Q-learning algorithm was developed in [19] for a joint computation offloading and resource allocation scheme in edge computing.…”

“…Moreover, this breaks the Markov properties required by the Q-learning algorithm and thus, DQN may not be capable of learning the cooperative offloading policies of EDs. Moreover, the non-cooperative multi-agent DRL solutions [17]- [19] may not be able to learn the cooperative policy; and thus resource usage over the edge network is not efficient which limits the overall offloading performance, e.g., offloading utility. • In addition, in most existing blockchain-based MEC schemes [22]- [26], the design and optimization of task offloading and blockchain mining are done separately, which would result in sub-optimal performance.…”

“…In the cooperative TOBM problem in our blockchain-based MEC system, conventional single-agent [13], [14], [32] or independent multi-agent [17]- [19] solutions are unable to obtain the cooperative policies of EDs due to the nonstationary and partially observable environment. Indeed, when policies of other agents change (i.e., due to computation mode preference), the ED (agent) n observation O n can be changed (nonstationary), which makes the obtained reward r n (t) different from the accumulated reward from its actual state-action pair.…”