Function Approximation Based Reinforcement Learning for Edge Caching in Massive MIMO Networks

Abstract

Caching popular contents in advance is an important technique to achieve low latency and reduced backhaul congestion in future wireless communication systems. In this article, a multi-cell massive multi-input-multi-output system is considered, where locations of base stations are distributed as a Poisson point process. Assuming probabilistic caching, average success probability (ASP) of the system is derived for a known content popularity (CP) profile, which in practice is time-varying and unknown in advance. Further, modeling CP variations across time as a Markov process, reinforcement Q-learning is employed to learn the optimal content placement strategy to optimize the long-term-discounted ASP and average cache refresh rate. In the Q-learning, the number of Q-updates are large and proportional to the number of states and actions. To reduce the space complexity and update requirements towards scalable Q-learning, two novel (linear and non-linear) function approximations-based Q-learning approaches are proposed, where only a constant (4 and 3 respectively) number of variables need updation, irrespective of the number of states and actions. Convergence of these approximation-based approaches are analyzed. Simulations verify that these approaches converge and successfully learn the similar best content placement, which shows the successful applicability and scalability of the proposed approximated Q-learning schemes.

Wireless Edge Caching and Content Popularity Prediction Using Machine Learning

“…RL enables the agent to learn to behave in an environment by performing actions and then analyzing the results [15]. The task of Q-learning is usually described as Markov Decision Process (MDP), however, state space, transition probabilities and reward functions are not required.…”

Ensemble Learning-Based Edge Caching Strategies for Internet of Vehicles: Outage and Finite SNR Analysis

“…Likewise, the content retrieval delay in cooperative cellular networks was analyzed in [15] while [16] characterized the successful content delivery performance for backhaullimited cache-enabled heterogeneous networks. In [17], the successful content delivery probability was analyzed for reinforcement learning-based probabilistic caching with a priori knowledge of content popularity. Similar successful content delivery probability analysis was also conducted in [18] for online content popularity prediction techniques.…”

A Survey on Reinforcement Learning-Aided Caching in Heterogeneous Mobile Edge Networks

“…In [83], a multi-cell multiple-input multiple-output (MIMO) system is studied where the locations of the BSs are modeled through a Poisson point process (PPP). Given a content popularity profile, the average success probability of the system can be derived under the probabilistic caching assumption.…”