Small basestations (SBs) equipped with caching units have potential to handle the unprecedented demand growth in heterogeneous networks. Through low-rate, backhaul connections with the backbone, SBs can prefetch popular files during off-peak traffic hours, and service them to the edge at peak periods. To intelligently prefetch, each SB must learn what and when to cache, while taking into account SB memory limitations, the massive number of available contents, the unknown popularity profiles, as well as the space-time popularity dynamics of user file requests. In this work, local and global Markov processes model user requests, and a reinforcement learning (RL) framework is put forth for finding the optimal caching policy when the transition probabilities involved are unknown. Joint consideration of global and local popularity demands along with cache-refreshing costs allow for a simple, yet practical asynchronous caching approach. The novel RL-based caching relies on a Q-learning algorithm to implement the optimal policy in an online fashion, thus enabling the cache control unit at the SB to learn, track, and possibly adapt to the underlying dynamics. To endow the algorithm with scalability, a linear function approximation of the proposed Q-learning scheme is introduced, offering faster convergence as well as reduced complexity and memory requirements. Numerical tests corroborate the merits of the proposed approach in various realistic settings.

No abstract

Abstract-Effective spectrum sensing strategies enable cognitive radios (CRs) to identify and opportunistically transmit on under-utilized spectral resources. In this paper, sequential channel sensing problems for single and multiple secondary users (SUs) networks are effectively modeled through finite state Markovian processes. More specifically, a model for single user case is introduced and its performance is validated through analytical analysis. Then, in order to address multiple SUs case, this model is extended to include the modified p-persistent access (MPPA) protocol. Since the scheme utilized experiences a high level of collision among the SUs, to mitigate the problem appropriately, p-persistent random access (PPRA) protocol is considered, which offers higher average throughput for SUs by statistically distributing their loads among all channels. The structure and performance of the proposed schemes are discussed in detail, and a set of illustrative numerical results is presented to validate and compare the performance of the proposed senseaccess strategies.Index Terms-Cognitive radio, spectrum handover, sequential channel sensing, queuing networks, secondary user's throughput.

Small base stations (SBs) of fifth-generation (5G) cellular networks are envisioned to have storage devices to locally serve requests for reusable and popular contents by caching them at the edge of the network, close to the end users. The ultimate goal is to shift part of the predictable load on the backhaul links, from on-peak to off-peak periods, contributing to a better overall network performance and service experience. To enable the SBs with efficient fetch-cache decision-making schemes operating in dynamic settings, this paper introduces simple but flexible generic time-varying fetching and caching costs, which are then used to formulate a constrained minimization of the aggregate cost across files and time. Since caching decisions per time slot influence the content availability in future slots, the novel formulation for optimal fetch-cache decisions falls into the class of dynamic programming. Under this generic formulation, first by considering stationary distributions for the costs and file popularities, an efficient reinforcement learningbased solver known as value iteration algorithm can be used to solve the emerging optimization problem. Later, it is shown that practical limitations on cache capacity can be handled using a particular instance of the generic dynamic pricing formulation. Under this setting, to provide a light-weight online solver for the corresponding optimization, the well-known reinforcement learning algorithm, Q-learning, is employed to find optimal fetchcache decisions. Numerical tests corroborating the merits of the proposed approach wrap up the paper.

No abstract