Two timescale convergent Q-learning for sleep-scheduling in wireless sensor networks

Abstract

In this paper, we consider an intrusion detection application for Wireless Sensor Networks. We study the problem of scheduling the sleep times of the individual sensors, where the objective is to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous stateaction spaces, in a manner similar to Fuemmeler and Veeravalli (IEEE Trans Signal Process 56(5), [2091][2092][2093][2094][2095][2096][2097][2098][2099][2100][2101] 2008). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation. Feature-based representations and function approximation is necessary to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation architecture for the Q-values) is updated in an on-policy temporal difference algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model and this is useful in settings where the latter is not known. Our simulation results on a synthetic 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work.

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

“…Specifically, the forward propagation algorithms apply the carefully trained weight matrixes and bias vectors for carrying out the associated linear and [294] reduced-state SARSA cellular network dynamic channel allocation considering both mobile traffic and call handoffs. [295] on-policy SARSA CR network distributed multiagent sensing policy relying on local interactions among SU [296] on-policy SARSA MANET energy-aware reactive routing protocol for maximizing network lifetime [297] on-policy SARSA HetNet resource management for maximizing resource utilization and guaranteeing QoS [298] approximate SARSA P2P network energy harvesting aided power allocation policy for maximizing the throughput [299] Q-learning WBAN power control scheme to mitigate interference and to improve throughput [300] Q-learning OFDM system adaptive modulation and coding not relying on off-line training from PHY [301] Q-learning cooperative network efficient relay selection scheme meeting the symbol error rate requirement [302] decentralized Q-learning CR network aggregated interference control without introducing signaling overhead [303] convergent Q-learning WSN sensors' sleep scheduling scheme for minimizing the tracking error activation operations. By contrast, the backward propagation algorithms, which are widely used in the industrial field define a so-called loss function for quantifying the difference between the output produced by the training samples' and the real output.…”

Intelligent Sensing and Computing in Wireless Sensor Networks for Multiple Target Tracking

“…Moreover, due to the extensive research on machine learning, lots of researches have upgraded and extended the algorithm, which can be widely used in WSNs [22]. In [23], they proposed a double time scale Q-learning algorithm with function approximation to alleviate the curse of dimension problems. Although all of the above algorithms alleviate the state explosion problem, it is necessary to solve the action explosion problem to obtain the scalable solution.…”

“…Otherwise, this action will be punished. Corresponding reward or penalty is responsible for adjusting weight parameters for the deep neural network [23].…”