Optimal and Scalable Caching for 5G Using Reinforcement Learning of Space-Time Popularities

Abstract

Small basestations (SBs) equipped with caching units have potential to handle the unprecedented demand growth in heterogeneous networks. Through low-rate, backhaul connections with the backbone, SBs can prefetch popular files during off-peak traffic hours, and service them to the edge at peak periods. To intelligently prefetch, each SB must learn what and when to cache, while taking into account SB memory limitations, the massive number of available contents, the unknown popularity profiles, as well as the space-time popularity dynamics of user file requests. In this work, local and global Markov processes model user requests, and a reinforcement learning (RL) framework is put forth for finding the optimal caching policy when the transition probabilities involved are unknown. Joint consideration of global and local popularity demands along with cache-refreshing costs allow for a simple, yet practical asynchronous caching approach. The novel RL-based caching relies on a Q-learning algorithm to implement the optimal policy in an online fashion, thus enabling the cache control unit at the SB to learn, track, and possibly adapt to the underlying dynamics. To endow the algorithm with scalability, a linear function approximation of the proposed Q-learning scheme is introduced, offering faster convergence as well as reduced complexity and memory requirements. Numerical tests corroborate the merits of the proposed approach in various realistic settings.

Reinforcement Learning Based Cooperative Coded Caching Under Dynamic Popularities in Ultra-Dense Networks

“…The grouped linear model is introduced in [30] to obtain the predicted content requests, based on which the cache replacement is optimized by using RL with model-free acceleration. In [31], a RL framework is proposed to obtain the optimal caching strategy at SBSs taking into account the space-time dynamics of the content popularity. In [32], the probabilistic caching strategy, resource allocation, and computation offloading at fog nodes are jointly considered to minimize the average transmission delay exploiting deep RL.…”

“…According to [44], the skewness of the Zipf-like distribution is dependent on the specific application. In brief, Zipf-like distribution is able to well depict the real user request distribution of various networks and thus is widely adopted in [18], [21], [25], [26], [31]. In this paper, we consider the mobile UDN and thus Zipf-like distribution can be safely used to model the user request distribution.…”

Online Convex Optimization for Caching Networks

“…Instead of fitting models, another option is to learn the popularity without using prior assumptions [21], [22]. For instance, [22] models the popularity evolution as a Markov process and employs Q-learning to estimate the transition probabilities which are then used for proactive caching. Such model-free solutions work well if there are adequate data, but have substantial computation and memory requirements.…”

Orchestration and Control in Software‐Defined 5G Networks: Research Challenges

“…In particular, researchers are working on Mobile Edge Computing (MEC) [94] and on what, where, when, and how to log cache contents. In this context, coding [95] and machine learning [96] techniques can be applied. Examples of typical scenarios are vehicular communications [97] and low-latency applications [98].…”