Reinforcement Learning Based Cooperative Coded Caching Under Dynamic Popularities in Ultra-Dense Networks

Abstract

For ultra-dense networks with wireless backhaul, caching strategy at small base stations (SBSs), usually with limited storage, is critical to meet massive high data rate requests. Since the content popularity profile varies with time in an unknown way, we exploit reinforcement learning (RL) to design a cooperative caching strategy with maximum-distance separable (MDS) coding. We model the MDS coding based cooperative caching as a Markov decision process to capture the popularity dynamics and maximize the long-term expected cumulative traffic load served directly by the SBSs without accessing the macro base station. For the formulated problem, we first find the optimal solution for a small-scale system by embedding the cooperative MDS coding into Q-learning. To cope with the large-scale case, we approximate the state-action value function heuristically. The approximated function includes only a small number of learnable parameters and enables us to propose a fast and efficient action-selection approach, which dramatically reduces the complexity. Numerical results verify the optimality/near-optimality of the proposed RL based algorithms and show the superiority compared with the baseline schemes. They also exhibit good robustness to different environments.

A Survey on Integrated Access and Backhaul Networks

“…The coded cooperation of cache-enabled IAB nodes was considered in (Gao et al, 2020) and (Vu et al, 2018). The authors in (Vu et al, 2018) studied the energy-efficiency performance of a cache-enabled IAB network, with the consideration of the cache capability.…”

“…Simulation results showed that uncoded caching outperforms coded caching in the small user cache size regime, while coded caching outperforms uncoded caching in the small BSs cache size regime. In (Gao et al, 2020), the authors applied the maximum distance separable (MDS) code at the IAB nodes. By leveraging the Q-learning model, the content caching decision is derived.…”

“…Optical wireless communication (OWC) using wavelength band ranges from 350 nm to 1550 nm is a promising alternative Gao et al (2020) Heuristic algorithm mmWave Ahangary et al (2020) solution to RF transmission (Ghassemlooy et al, 2015). This section reviews the studies for the incorporation of visible light communication (VLC) and free-space optical (FSO) communication into the IAB networks, respectively.…”

A Survey of Deep Learning for Data Caching in Edge Network

“…A DQN is proposed for finding a suboptimal solution. Though Reference [89] is regarded as solving what to cache problem, like Reference [79][80][81][82][83][84][85][86][87][88]102], the reinforcement learning approach plays a different role. In Reference [89], a DNN is utilized for content popularity prediction and then an RL is used for DNN hyperparameters tuning instead of determining caching content.…”

“…Reference [93] provides a DDPG model to cope with continuous valued control decision for 3C in vehicular edge networks. The work in Reference [102] focuses on the cooperative caching policy at FBSs with maximum distance separable coding in ultra dense networks. A Q-learning model is utilized to determine caching categories and the content quantity at FBSs during the off-peak duration.…”