Collaborative Edge Computing and Caching With Deep Reinforcement Learning Decision Agents

Ren, Jianji; Hou, Tingting; Zheng, Shuai; Tang, Chaosheng

doi:10.1109/access.2020.3007002

Cited by 28 publications

(9 citation statements)

References 22 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Through a review on literature one can find three different approaches. First, there is research targeting specific topics of these tiers, such as data caching [2], [3], service deployment [4], computation offloading [5], [6], resource allocation [7], scheduling [8], [9], traffic grooming [10] or service decomposition [11], among others. Second, there is research that addresses these specific topics but they are contextualized in a specific domain.…”

Section: Related Workmentioning

confidence: 99%

On Distributed Computing Continuum Systems

Dustdar

Pujol

Donta

2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

This article presents our vision on the need of developing new managing technologies to harness distributed "computing continuum" systems. These systems are concurrently executed in multiple computing tiers: Cloud, Fog, Edge and IoT. This simple idea develops manifold challenges due to the inherent complexity inherited from the underlying infrastructures of these systems. This makes inappropriate the use of current methodologies for managing Internet distributed systems, which are based on the early systems that were based on client/server architectures and were completely specified by the application software. We present a new methodology to manage distributed "computing continuum" systems. This is based on a mathematical artifact called Markov Blanket, which sets these systems in a Markovian space, more suitable to cope with their complex characteristics. Furthermore, we develop the concept of equilibrium for these systems, providing a more flexible management framework compared with the one based on thresholds, currently in use for Internet-based distributed systems. Finally, we also link the equilibrium with the development of adaptive mechanisms. However, we are aware that developing the entire methodology requires a big effort and the use of learning techniques, therefore, we finish this article with an overview of the techniques required to develop this methodology.

show abstract

Section: Related Workmentioning

confidence: 99%

On Distributed Computing Continuum Systems

Dustdar

Pujol

Donta

2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…A slightly different problem is considered in [116] where the capability of users to offload computing tasks to edge computing nodes is examined. In this context, the coordination between edge computing nodes for the management of the compute and cache resources is investigated.…”

Section: ) Delay Reductionmentioning

confidence: 99%

“…However, accuracy, scalability and efficiency in HetNets can be further improved through real-time heuristics and analytics. In a different topology, the solution in [116] offloads tasks to edge computing nodes, investigating the management strategy of the compute and cache resources. In this area, further research on the use of competitive bidding and allocation priorities can enable additional gains.…”

Section: F Cooperative Caching Extensionsmentioning

confidence: 99%

A Survey on Reinforcement Learning-Aided Caching in Heterogeneous Mobile Edge Networks

et al. 2022

View full text Add to dashboard Cite

Mobile networks experience a tremendous increase in data volume and user density due to the massive number of coexisting users and devices. An efficient technique to alleviate this issue is to bring the data closer to the users by exploiting cache-aided edge nodes, such as fixed and mobile access points, and even user devices. Meanwhile, the fusion of machine learning and wireless networks offers new opportunities for network optimization when traditional optimization approaches fail or incur high complexity. Among the various machine learning categories, reinforcement learning provides autonomous operation without relying on large sets of historical data for training. In this survey, reinforcement learning-aided mobile edge caching solutions are presented and classified, based on the networking architecture and optimization target. As sixth generation (6G) networks will be characterized by high heterogeneity, fixed cellular, fog, cooperative, vehicular, and aerial networks are studied. The discussion of these works reveals that there exist reinforcement learning-aided caching schemes with varying complexity that can surpass the performance of conventional policy-based approaches. Finally, several open issues are presented, stimulating further interest in this important research field.INDEX TERMS 6G, edge caching, heterogeneous networks, machine learning, mobile edge networks, reinforcement learning.

show abstract

“…A DRL-based caching strategy was proposed in [11] to improve the efficiency of the cache content when the content popularity is dynamic and unknown by using a deep Q neural network to approximate the Q action value function and optimizing parameters of the network. In order to improve the long-term profit of MEC and meet the low latency requirements of users, a joint optimization strategy with offloading calculation and resource allocation based on a DDQN was proposed to improve service quality in [12]. A distributed algorithm based on long short-term memory, the Dueling Deep Q Network (D2QN) and the Double Deep Q Network (DDQN) was proposed in [13] to determine the offloading decision without knowing the task model for minimizing the long-term average cost of the system.…”

Section: Introductionmentioning

confidence: 99%

Joint Edge Computing and Caching Based on D3QN for the Internet of Vehicles

et al. 2023

View full text Add to dashboard Cite

With the Internet of Vehicles (IOV), a lot of self-driving vehicles (SDVs) need to handle a variety of tasks but have very seriously limited computing and storage resources, meaning they cannot complete intensive tasks timely. In this paper, a joint edge computing and caching based on a Dueling Double Deep Q Network (D3QN) is proposed to solve the problem of the multi-task joint edge calculation and caching process. Firstly, the processes of offloading tasks and caching them to the base station are modeled as optimization problems to maximize system revenues, which are limited by system latency and energy consumption as well as cache space for computing task constraints. Moreover, we also take into account the negative impact of the number of unfinished tasks in relation to the optimization problem—the higher the number of unfinished tasks, the lower the system revenue. Secondly, we use the D3QN algorithm together with the cache models to solve the formulated NP-hard problem and select the optimal caching and offloading action by adopting an e-greedy strategy. Moreover, two cache models are proposed in this paper to cache tasks, namely the active cache, based on the popularity of the task, and passive cache, based on the D3QN algorithm. Additionally, tasks which deal with cache space are updated by computing the expulsion value based on type of popularity. Finally, simulation results show that the proposed algorithm has good performance in terms of the latency and energy consumption of the system and that it improves utilization of cache space and reduces the probability of unfinished tasks. Compared to the Deep Q Network with caching policy, with the Double Deep Q Network with caching policy and Dueling Deep Q Network with caching policy, the system revenue of the proposed algorithm is improved by 65%, 35% and 66%, respectively. The scenario of the IOV proposed in this article can be expanded to larger-scale IOV systems by increasing the number of SDVs and base stations, and the content caching and download functions of the Internet of Things can also be achieved through collaboration between multiple base stations. However, only the cache model is focused on in this article, and the design of the replacement model is not good enough, resulting in a low utilization of cache resources. In future work, we will analyze how to make joint decisions based on multi-agent collaboration for caching, offloading and replacement in IOV scenarios with multiple heterogeneous services to support different Vehicle-to-Everything services.

show abstract

Collaborative Edge Computing and Caching With Deep Reinforcement Learning Decision Agents

Cited by 28 publications

References 22 publications

On Distributed Computing Continuum Systems

On Distributed Computing Continuum Systems

A Survey on Reinforcement Learning-Aided Caching in Heterogeneous Mobile Edge Networks

Joint Edge Computing and Caching Based on D3QN for the Internet of Vehicles

Contact Info

Product

Resources

About