RL-based superframe order adaptation algorithm for IEEE 802.15.4 networks

Abstract

In wireless sensor networks, it is an important problem to adjust the work time window in each working/sleeping period to save energy under light network loads and decrease the packet delay under heavy network loads. In this paper, we introduce reinforcement learning method into this problem. We discuss the algorithm design method in a simple IEEE 802.15.4 network, where an RL-based adaptive algorithm is proposed. Simulation results show that this RL-based algorithm can adapt to the change of data flow and make a good tradeoff between the energy-saving performance and the packet delay performance.

Two timescale convergent Q-learning for sleep-scheduling in wireless sensor networks

“…(iii) In comparison to the RL based approaches [18,22,23] for transmission scheduling at the MAC layer, we would like to point out that the algorithms proposed there (a) employ full state representations; (b) consider discrete state-action spaces (except [23] which adapts Qlearning for continuous actions, albeit with a discrete state space); (c) consider an MDP with perfect information, i.e., a setting where the states are fully observable; (d) consider only a discounted setting, which is not amenable for studying steady state system behaviour; (e) are primarily concerned with managing transmission in an energy-efficient manner and not with tracking an intruder with highaccuracy. In other words, the algorithms of [18,22,23] are not applicable in our setting as we consider a partially observable MDP with continuous state-action spaces, and 1 A short version of this paper containing only the average cost setting and algorithms and with no proofs is available in [24]. The current paper includes in addition: (i) algorithms for the discounted cost setting; (ii) a detailed proof of convergence of the average cost algorithm using theory of stochastic recursive inclusions; and (iii) detailed numerical experiments.…”

“…(iv) Many RL based approaches proposed earlier for sleep scheduling (see [18,22,23,29]) employ full state representations and hence, they are not scalable to larger networks owing to the curse of dimensionality. We employ efficient linear approximators to alleviate this.…”

“…The algorithms proposed there attempt to maximize the throughput while being energy efficient. In [22,23], the authors propose Q-learning based algorithms, whereas, in [18], the authors propose an algorithm based on SARSA. In [16], the authors present two sleep scheduling algorithms for single object tracking.…”

“…In (18), the action a n is chosen in state s n according to an -greedy policy, i.e., with a probability of (1 À ), a greedy action given by a n ¼ argmin v2Aðs n Þ h T n r s n ;v is chosen and with probability , an action in Aðs n Þ is randomly chosen. Using -greedy policy for the regular Q-learning algorithm has been well recognized and recommended in the literature (cf.…”

“…In [18,22,23], the authors propose RL based medium access control (MAC) protocols for WSNs. The algorithms proposed there attempt to maximize the throughput while being energy efficient.…”