Multi-Agent Reinforcement Learning Based Cooperative Content Caching for Mobile Edge Networks

Abstract

Designing clustered unmanned aerial vehicle (UAV) communication networks based on cognitive radio (CR) and reinforcement learning can significantly improve the intelligence level of clustered UAV communication networks and the robustness of the system in a time-varying environment. Among them, designing smarter systems for spectrum sensing and access is a key research issue in CR. Therefore, we focus on the dynamic cooperative spectrum sensing and channel access in clustered cognitive UAV (CUAV) communication networks. Due to the lack of prior statistical information on the primary user (PU) channel occupancy state, we propose to use multiagent reinforcement learning (MARL) to model CUAV spectrum competition and cooperative decision-making problem in this dynamic scenario, and a return function based on the weighted compound of sensing-transmission cost and utility is introduced to characterize the real-time rewards of multi-agent game. On this basis, a time slot multi-round revisit exhaustive search algorithm based on virtual controller (VC-EXH), a Q-learning algorithm based on independent learner (IL-Q) and a deep Q-learning algorithm based on independent learner (IL-DQN) are respectively proposed. Further, the information exchange overhead, execution complexity and convergence of the three algorithms are briefly analyzed. Through the numerical simulation analysis, all three algorithms can converge quickly, significantly improve system performance and increase the utilization of idle spectrum resources.

Joint Trajectory-Task-Cache Optimization in UAV-Enabled Mobile Edge Networks for Cyber-Physical System

“…Then r k is constrained to r k ≤ R k so as to guarantee GT task U k executed before its completion deadline. Then one has (29) where the inequality is due to the fact that the UAV will execute each offloaded task quicker than the task being conducted by GT locally, i.e. a k F k…”

Analysis of Data Persistence in Collaborative Content Creation Systems: The Wikipedia Case

“…Typical examples in this sense are web caching systems that store web content in user browsers or in dedicated http proxies. Today, caching is proposed to improve the efficiency of novel mobile edge computing solutions [9] or to optimize content dissemination in opportunistic networks where device-to-device connection is time limited and unpredictable [48,49]. In information-centric networking, a recent paradigm that moves the focus of networking from host-to-host communication to content-oriented delivery, caching is an essential element of the network since all nodes can store content objects in an internal memory.…”

“…Content update statistics evolved to become a fundamental component in the design and study of any communication network where caching and prefetching play a central role, such as content delivery networks (CDNs) and Information-centric networks (ICNs) [3][4][5], 5G networks [6][7][8][9], and wireless networks [10]. Indeed, accurate estimation of inter-update distribution is an important issue.…”

Joint Optimization of Caching and Computation in Multi-Server NOMA-MEC System via Reinforcement Learning

“…In addition, for caching, we need to consider task popularity, which is an important factor affecting the caching. Task popularity refers to the probability of a task being requested within a specific time [24]. Most articles believe that popularity is known and follows a Zipf distribution [25]- [28].…”

“…Most articles believe that popularity is known and follows a Zipf distribution [25]- [28]. However, in the actual scenario, due to the dynamics of the task and the mobility of the user, the popularity changes dynamically over time, so it can be considered as a time series [24]. Gated Recurrent Unit (GRU) algorithm is a method of deep learning, which can deal with vanishing gradient and efficiently capture long-term dependencies [29].…”