Deep Reinforcement Learning for Energy Efficiency Maximization in Cache-Enabled Cell-Free Massive MIMO Networks: Single- and Multi-Agent Approaches

Abstract

Cell-free massive multiple-input multiple-output (CF-mMIMO) is an emerging beyond fifth-generation (5G) technology that improves energy efficiency (EE) and removes cell structure limitation by using multiple access points (APs). This study investigates the EE maximization problem. Forming proper cooperation clusters is crucial when optimizing EE, and it is often done by selecting AP-user pairs with good channel quality or aligning AP cache contents with user requests. However, the result can be suboptimal if we determine the clusters based solely on either aspect. This motivates our joint design of user association and content caching. Without knowing the user content preferences in advance, two deep reinforcement learning (DRL) approaches, i.e., single-agent reinforcement learning (SARL) and multi-agent reinforcement learning (MARL), are proposed for different scenarios. The SARL approach operates in a centralized manner which has lower computational requirements on edge devices. The MARL approach requires more computation resources at the edge devices but enables parallel computing to reduce the computation time and therefore scales better than the SARL approach. The numerical analysis shows that the proposed approaches outperformed benchmark algorithms in terms of network EE in a small network. In a large network, the MARL yielded the best EE performance and its computation time was reduced significantly by parallel computing.

An Architecture for AoI and Cache Hybrid Multicast/Unicast/D2D With Cell-Free Massive MIMO Systems

“…Driven by the observation, extensive study has been made in researching the benefits of caching in the offloading and the throughput performance with various caching strategies offered. For further information, see [22], [23], and [24] and references therein.…”

“…In comparison to the cellular counterpart, the abundance of spatial diversity gain and extraordinary macroscopic diversity provided by the advent of cell-free massive multiple-input multipleoutput (MIMO) have been regarded as a new paradigm in the design to effectively handle the rapid growth of data traffic, resolve access collision, and enhance capacity with affordable complexity [7], [8], [9], [10], which opens up new avenues for the enhanced mMTC, uRLLC, and eMBB. In addition, significant breakthroughs including software-defined networking (SDN) [11], [12], network function virtualization (NFV) [12], [13], multicast transmission [14], [15], [16], [17], [18], device-to-device (D2D) [19], [20], [21], wireless cache [22], [23], [24], and age of information (AoI) [25], [26], [27], [28], [29] have also been recognised as a main building blocks technology as well. Especially, the typical traffics of eMBB, mMTC, and uRLLC for main 6G technology in presented in Fig.…”

Learning-Based Joint User Association and Cache Replacement for Cache-Enabled Cloud RAN

“…In addition to the content placement, the user association also should be carefully decided by taking account of BS-user link conditions and cache status at BSs jointly to strike a balance between the cache hit ratio and communication reliability [22], [23]. In this context, the joint optimization of content placement and user association have been tackled with iterative algorithms [24]- [29], latent factor model (LFM) [30], and deep reinforcement learning (DRL) [31], [32].…”

“…Although the joint optimization with user association has potential to boost the edge caching gain by expanding available caches for serving a user, similarly as in the case of content placement [24]- [28], [31], [32], the conventional work on the cache replacement have not considered the cacheaware flexible user association. Specifically, most existing studies on cache replacement have considered fixed cellular regions and corresponding user associations, so that their cache replacement strategies have concentrated on cooperative edge caching for given user association [34], [36]- [38].…”

Cluster Content Caching: A Deep Reinforcement Learning Approach to Improve Energy Efficiency in Cell-Free Massive Multiple-Input Multiple-Output Networks

“…Therefore, its total energy efficiency (EE) maximization problem is non-deterministic polynomial-hard (NP-hard) and necessitates solutions through inefficient and non-scalable methods. Additionally, researchers have started to consider the joint optimization of user association and caching strategies [19][20][21]. For example, in [19], the high-density satellite-UAV-terrestrial network scenario is considered, and the initial combination optimization problem is effectively solved using game theory and genetic algorithm for clustering and cache placement, respectively.…”