IRS Assisted NOMA Aided Mobile Edge Computing With Queue Stability: Heterogeneous Multi-Agent Reinforcement Learning

Abstract

By employing powerful edge servers for data processing, mobile edge computing (MEC) has been recognized as a promising technology to support emerging computation-intensive applications. Besides, non-orthogonal multiple access (NOMA)aided MEC system can further enhance the spectral-efficiency with massive tasks offloading. However, with more dynamic devices brought online and the uncontrollable stochastic channel environment, it is even desirable to deploy appealing technique, i.e., intelligent reflecting surfaces (IRS), in the MEC system to flexibly tune the communication environment and improve the system energy efficiency. In this paper, we investigate the joint offloading, communication and computation resource allocation for the IRS-assisted NOMA MEC system. We first formulate a mixed integer energy efficiency maximization problem with system queue stability constraint. We then propose the Lyapunovfunction-based Mixed Integer Deep Deterministic Policy Gradient (LMIDDPG) algorithm which is based on the centralized reinforcement learning (RL) framework. To be specific, we design the mixed integer action space mapping which contains both continuous mapping and integer mapping. Moreover, the award function is defined as the upper-bound of the Lyapunov driftplus-penalty function. To enable end devices (EDs) to choose actions independently at the execution stage, we further propose the Heterogeneous Multi-agent LMIDDPG (HMA-LMIDDPG) algorithm based on distributed RL framework with homogeneous EDs and heterogeneous base station (BS) as heterogeneous multiagent. Numerical results show that our proposed algorithms can achieve superior energy efficiency performance to the benchmark algorithms while maintaining the queue stability. Specially, the distributed structure HMA-LMIDDPG can acquire more energy efficiency gain than the centralized structure LMIDDPG.

6G Mobile-Edge Empowered Metaverse: Requirements, Technologies, Challenges and Research Directions

“…The size of IRS elements is 64, and there are 4 users in the NOMA sub-carrier cluster. Details about the system and configurations can be found in [12]. Centralized RL-supported MEC refers to the agent deployment at the edge server, whereas distributed DTassisted MEC refers to the heterogeneous agents deployment on both edge server and user devices.…”

Towards Energy-Efficiency: Integrating DRL and ze-RIS for Task Offloading in UAV-MEC Environments

“…In association to it, a waveform design and detection mechanism in NOMA systems is also presented in [28]. Work presented in [29] also considered a IRS based NOMA assisted MEC mechanism for energy efficiency with queue stability option. A Lyapunovfunction-based mixed integer DDPG (LMIDDPG) algorithm is proposed for centralized learning and a heterogeneous MA-LMIDDPG algorithm is proposed for distributed learning, considering system throughput, power consumption, and queue stability for computing decisions, power allocation, and phase shifts optimizations.…”

Communication-Assisted Multi-Agent Reinforcement Learning Improves Task-Offloading in UAV-Aided Edge-Computing Networks

“…Mobile edge computing (MEC) is a key technique of improving the quality of experience (QoE) of mobile users by offloading the computation tasks from users [1,2]. Typically, MEC servers are deployed at fixed locations near the wireless edge, limiting their capability of providing flexible services.…”

“…In order to obtain a decentralized policy, Hernandez-Leal et al [12] and Yu et al [2] employed a distributed learner for each MEC agent, which however may result in a nonstationary problem [12,14]. To tackle this problem, Wang et al in [13] resorted to a centralized training and decentralized execution (CTDE) framework to control a UAV's trajectory, aiming at maximizing the offloading fairness, while minimizing the overall energy consumption.…”

“…Select action an from based on on,t and Mn,t using ϵgreedy Qn(on,t, an,t; θ) for agent n = 1 to N . Compute the critic loss as 2 , where y i = r i + γ maxa Qtot(o ′i , a; θ). For the channel model, we set α = 2, µ = −3 dB as in [21].…”