Q-Routing in Cognitive Packet Network Routing Protocol for MANETs

Abstract

Mobile Ad hoc Networks (MANET) are self-organized networks which are characterized by dynamic topologies in time and space. This creates an instable environment, where classical routing approaches cannot achieve high performance. Thus, adaptive routing is necessary to handle the random changing network topology. This research uses Reinforcement Learning approach with Q-Routing to introduce our MANET routing algorithm: Stability-Aware Cognitive Packet Network (CPN). This new algorithm extends the work on CPN to adapt it to the MANET environment with focus on path stability metric. CPN is a distributed adaptive routing protocol that uses three types of packets: Smart Packets for route discovery, Data Packets for carrying data payload, and Acknowledgments to bring back feedback information for the Reinforcement Learning reward function. The research defines a reward function as a combination of high stability and low delay path criteria to discover long-lived routes without disrupting the overall delay. The algorithm uses Acknowledgment-based Q-routing to make routing decisions which adapt on line to network changes allowing nodes to learn efficient routing policies.

Scalable Deep Reinforcement Learning for Routing and Spectrum Access in Physical Layer

“…We can then re-estimate the values of Q π (s t , a t ) through either ( 13) or (14), and continue to improve the policy using (15). When the process converges, we reach an optimal policy π * (and its corresponding value functions Q π * (s t , a t )).…”

“…When estimating Q (which is commonly referred to as policy evaluation), there are two common approaches, each with its own benefits and shortcomings: 1) Monte-Carlo Estimation, corresponding to (13), which provides a unbiased estimation, but has high variance; 2) Temporal Difference Estimation, corresponding to (14), which has low variance, but is biased due to its self-referential nature.…”

“…To test the generalizability of the proposed method, we directly take the agent trained under the original setting and reuse it for a much larger ad-hoc network of F = 10 data flows in a 5000×5000m 2 region, with B = 32 available frequency bands. We place larger number of (19,16,21,18,14,24,17,20,19) nodes over nine evenly divided sub-regions. The sum-rate results are shown in Fig.…”

Scalable Reinforcement Learning For Routing In Ad-Hoc Networks Based On Physical-Layer Attributes

“…Thus, routing can be readily modeled as a Markov decision process [7] and naturally fits into the realm of reinforcement learning. In this direction, many previous works [8,9,10,11,12,13,14,15,16] have employed the classical Q-learning [17] algorithm to train agents to find the optimal route. In these works, a distinct agent is associated with each transmission node.…”