Resource Pricing and Allocation in MEC Enabled Blockchain Systems: An A3C Deep Reinforcement Learning Approach

Du, Jianbo; Cheng, Wenjie; Lu, Guangyue; Cao, Haotong; Chu, Xiaoli; Zhang, Zhicai; Wang, Junxuan

doi:10.1109/tnse.2021.3068340

“…Hybrid DRL method: In [25], The task offloading method was proposed based on meta reinforcement learning, which can adapt fast to new environments with a small number of gradient updates and samples. In [26], an A3C deep reinforcement learning algorithm was introduced to obtain the resource pricing and allocation MEC enabled blockchain systems. In [27], the DRL with Monte Carlo tree learning was introduced to solve the complex resource allocation problem for a collaborative MEC network.…”

Section: Related Workmentioning

confidence: 99%

Distributed Resource Scheduling for Large-Scale MEC Systems: A Multiagent Ensemble Deep Reinforcement Learning With Imitation Acceleration

Jiang

¹

,

Dong

²

,

Wang

³

et al. 2022

IEEE Internet Things J.

View full text Add to dashboard Cite

In large-scale mobile edge computing (MEC) systems, the task latency and energy consumption are important for massive resource-consuming and delay-sensitive Internet of things devices (IoTDs). Against this background, we propose a distributed intelligent resource scheduling (DIRS) framework to minimize the sum of task latency and energy consumption for all IoTDs, which can be formulated as a mixed integer nonlinear programming. The DIRS framework includes centralized training relying on the global information and distributed decision making by each agent deployed in each MEC server. Specifically, we first introduce a novel multi-agent ensemble-assisted distributed deep reinforcement learning (DRL) architecture, which can simplify the overall neural network structure of each agent by partitioning the state space and also improve the performance of a single agent by combining decisions of all the agents. Secondly, we apply action refinement to enhance the exploration ability of the proposed DIRS framework, where the near-optimal state-action pairs are obtained by a novel Levy flight search. Finally, an imitation acceleration scheme is presented to pre-train all the agents, which can significantly accelerate the learning process of the proposed framework through learning the professional experience from a small amount of demonstration data. The simulation results in three typical scenarios demonstrate that the proposed DIRS framework is efficient and outperforms the existing benchmark schemes.

show abstract

“…In mobile networks, such as social networks [43]- [45] and blockchain-based networks [82], [98], [104], [178]- [182], MEC is beneficial to trust-based communication among mobile devices due to its avoidance of data offloading to the central cloud and its capability of low-delay edge computing. In MEC-enabled mobile social networks with caching and D2D communication, according to social relationships, a trust-based resource allocation mechanism is devised [43]- [45].…”

Section: E Trust-based Applicationsmentioning

confidence: 99%

“…In [182], to maximize the system utility of all miners, A3Cbased resource allocation and pricing were proposed. The system utility is defined as the difference in the price-based probability minus the sum of the transmission price and computational price.…”

Section: E Trust-based Applicationsmentioning

confidence: 99%

Reinforcement Learning-Empowered Mobile Edge Computing for 6G Edge Intelligence

Wei

¹

,

Guo

²

,

Li

³

et al. 2022

View full text Add to dashboard Cite

Mobile edge computing (MEC) is considered a novel paradigm for computation-intensive and delay-sensitive tasks in fifth generation (5G) networks and beyond. However, its uncertainty, referred to as dynamic and randomness, from the mobile device, wireless channel, and edge network sides, results in high-dimensional, nonconvex, nonlinear, and NP-hard optimization problems. Thanks to the evolved reinforcement learning (RL), upon iteratively interacting with the dynamic and random environment, its trained agent can intelligently obtain the optimal policy in MEC. Furthermore, its evolved versions, such as deep reinforcement learning (DRL), can achieve higher convergence speed efficiency and learning accuracy based on the parametric approximation for the large-scale state-action space. This paper provides a comprehensive research review on RL-enabled MEC and offers insight for development in this area. More importantly, associated with free mobility, dynamic channels, and distributed services, the MEC challenges that can be solved by different kinds of RL algorithms are identified, followed by how they can be solved by RL solutions in diverse mobile applications. Finally, the open challenges are discussed to provide helpful guidance for future research in RL training and learning MEC. INDEX TERMS Mobile edge computing (MEC), network uncertainty, reinforcement learning (RL).

show abstract

“…In mobile networks, such as social networks [42]- [44] and blockchain-based networks [77], [93], [99], [175]- [179],…”

Section: E Trust-based Applicationsmentioning

confidence: 99%

“…To maximize the longterm payoff of the service provider and miners, a hierarchical RL is designed by combining the RL-based offloading strategy for miners with the RL-based price assignment for the service provider. In [179], to maximize the system utility of all miners, A3C-based resource allocation and pricing were proposed.…”

Section: E Trust-based Applicationsmentioning

confidence: 99%

Reinforcement Learning-Empowered Mobile Edge Computing for 6G Edge Intelligence

Wei,

Guo,

Li

et al. 2022

Preprint

0

View full text Add to dashboard Cite

Mobile edge computing (MEC) is considered a novel paradigm for computation-intensive and delay-sensitive tasks in fifth generation (5G) networks and beyond. However, its uncertainty, referred to as dynamic and randomness, from the mobile device, wireless channel, and edge network sides, results in high-dimensional, nonconvex, nonlinear, and NP-hard optimization problems. Thanks to the evolved reinforcement learning (RL), upon iteratively interacting with the dynamic and random environment, its trained agent can intelligently obtain the optimal policy in MEC. Furthermore, its evolved versions, such as deep RL (DRL), can achieve higher convergence speed efficiency and learning accuracy based on the parametric approximation for the large-scale state-action space. This paper provides a comprehensive research review on RL-enabled MEC and offers insight for development in this area. More importantly, associated with free mobility, dynamic channels, and distributed services, the MEC challenges that can be solved by different kinds of RL algorithms are identified, followed by how they can be solved by RL solutions in diverse mobile applications. Finally, the open challenges are discussed to provide helpful guidance for future research in RL training and learning MEC. Index Terms-Mobile edge computing (MEC), network uncertainty, reinforcement learning (RL). emote Areas irplane taverse vers Diverse Scenarios MEC Deployment Future Applications Smart City Smart Home Hazard/Remote Areas Smart Agriculture Digital Twin Holographic Communication AR/VR/XR Metaverse Metaverse Metavers Autonomous Driving Small Data Center Switch BS Traffic Light Satellite Ship Airplane S ll S it h BS T ffi Li ht S t llit Shi Ai l 5G Networks and Beyond 5G Networks and Beyond Smart Factory performance. Its typical algorithms are K-means clustering, principal component analysis, and independent component analysis, which can be used in small cell clustering, heterogeneous network clustering, smart grid user classification, etc. However, in unreliable radio network environments, the classification accuracy decreases, which readily causes slow and inaccurate actions in MEC systems. • Different from the static solutions provided by supervised learning and unsupervised learning, RL gives a constantly evolving intelligent framework [21]-[27]. In RL [29], an agent is enabled to make proper decisions based on frequent interactions with the stochastic and dynamic environment. Based on the Markov models, the RL operates in a feedback mechanism (a closed loop) without the knowledge of input data, where based on the previous and current states, the agent can execute its actions to maximize the reward function. Additionally, the above behaviors, including the states, actions, and rewards, accumulate to generate experiences. The classic RL algorithm is Q-learning, which suffers from the curse of dimensionality caused by the large dimensions of the state-action spaces. Accordingly, upon leveraging the low-dimensional representation for the high-dimensional state-ac...

show abstract

Resource Pricing and Allocation in MEC Enabled Blockchain Systems: An A3C Deep Reinforcement Learning Approach

Cited by 106 publications

References 38 publications

Distributed Resource Scheduling for Large-Scale MEC Systems: A Multiagent Ensemble Deep Reinforcement Learning With Imitation Acceleration

Distributed Resource Scheduling for Large-Scale MEC Systems: A Multiagent Ensemble Deep Reinforcement Learning With Imitation Acceleration

Reinforcement Learning-Empowered Mobile Edge Computing for 6G Edge Intelligence

Reinforcement Learning-Empowered Mobile Edge Computing for 6G Edge Intelligence

Contact Info

Product

Resources

About