A Reinforcement Learning Approach for the Multichannel Rendezvous Problem

Abstract

In this paper, we consider the multichannel rendezvous problem in cognitive radio networks (CRNs) where the probability that two users hopping on the same channel have a successful rendezvous is a function of channel states. The channel states are modelled by two-state Markov chains that have a good state and a bad state. These channel states are not observable by the users. For such a multichannel rendezvous problem, we are interested in finding the optimal policy to minimize the expected time-to-rendezvous (ETTR) among the class of dynamic blind rendezvous policies, i.e., at the t th time slot each user selects channel i independently with probability pi(t), i = 1, 2, . . . , N . By formulating such a multichannel rendezvous problem as an adversarial bandit problem, we propose using a reinforcement learning approach to learn the channel selection probabilities pi(t), i = 1, 2, . . . , N . Our experimental results show that the reinforcement learning approach is very effective and yields comparable ETTRs when comparing to various approximation policies in the literature.

Copula-Based Bounds for Multi-User Communications -- Part I: Average Performance

“…Statistical dependency has an important impact in many more applications. These include cognitive radio [45], queuing [46], and improper signaling [47]. A more general dependency analysis using copulas might also be beneficial in the context of finite-length information theory.…”

Enhanced Pub/Sub Communications for Massive IoT Traffic with SARSA Reinforcement Learning

“…An evaluation of SARSA and various other model-free algorithms is presented in Dafazio [9]. The evaluation considered diverse and difficult problems within a consistent environment (Arcade [15]) involving configuration aspects like Epsilon-greedy policy, exploration versus exploitation and state space with SARSA outperforming algorithms like Q-learning, ETTR (Expected Time-to-Rendezvous) [34], R-learning [16], GQ algorithm [35] and Actor-Critic algorithm [6].…”

An efficient coprime integer sets based rendezvous algorithm for distributed cognitive radio networks

“…Introduction: Multichannel blind rendezvous problem refers to how to guarantee that two secondary users (or called users for simplicity) hop on the same channel at the same timeslot in cognitive radio networks (CRNs) [1]. A number of works have been conducted on how to design channel hopping (CH) sequences for this fundamental problem in CRNs [2].…”

“…A number of works have been conducted on how to design channel hopping (CH) sequences for this fundamental problem in CRNs [2]. Existing works can be categorized from different perspectives as follows: (1) According to whether users can adopt the same CH strategy, they are divided into Asymmetric role [3][4][5] and Symmetric role [6]; (2) According to the status of available channels, they can be divided into Homogeneous channels [7], Heterogeneous channels [8], Channel invariant [9], and Channel variable [10]; (3) According to whether users start CH at the same time, they can be divided into Asynchronous clock [11,12] and Synchronous clock [13]; (4) According to one or more users receiving information, they can be divided into Pairwise destination (Unicast) [14] and Multiple destination (Multicast) [15]. The ideal CH algorithms are expected to give good performance in terms of the maximum time-to-rendezvous (MTTR) (i.e.…”

“…to denote the set of locally available channels to user A and user B, respectively. We assume that there is at least one common available channel between users A and B, that is, (1), S B (2), …, S B (T B )] denote the CH sequences of user A and user B within one cycle, respectively, where T A and T B represent the period lengths of their CH sequences. In the asynchronous clock scenario, we suppose user A starts its CH with δ (δ ≥ 0) timeslots earlier than user B.…”