Effective utilization of mobile ad hoc underwater distributed networks is challenging due to high system costs and the harsh environment characterized by low bandwidth, large latency, high energy consumption, and node mobility. This work addresses the routing issue, which is critical in successfully establishing and utilizing an underwater network. In particular, it focuses on reinforcement learning (RL)-based routing algorithms, which possess the ability to explore the network environment and adapt routing decisions to the constantly changing topology of the network due to node mobility and energy usage. This paper presents a routing algorithm based on Q-learning, one of the RL approaches, with additional Kinematic and Sweeping features, therefore referred to as QKS. These two additional features are introduced to address the potential slow convergence associated with pure RL algorithms. The results of a detailed packet-level simulation have been obtained using the NS-2 open-source network simulator with underwater modeling additions. The energy efficiency, convergence, and delivery performance of QKS are compared with two other routing protocols for underwater networks, a basic flooding approach (ICRP (Liang, 2007)) and a basic Q-learning implementation (QELAR (Hu, 2010)), using simulations of networks with both fixed and mobile nodes.
Distributed multistatic active sonar networks provide an Anti-Submarine Warfare capability against small, quiet, threat submarines in the harsh clutter-saturated littoral and deeper ocean environments. Adaptive ping control techniques provide the potential to significantly increase the multistatic network's performance, by pinging (in an optimum sense) the right source, at the right time, with the right waveform. This paper describes an automatic, adaptive ping control algorithm. It specifically addresses the "trackhold" objective, which is to adapt multistatic sonar operations to maintain and hold one or more target tracks which have been previously initiated (detected). The approach is unique in that it includes both sonar performance modeling and multistatic tracker outputs, in a closed-loop control structure. The paper motivates the approach, describes the algorithm, and shows some validating results. The evaluation utilizes a simple sonar performance model, a ping contact simulator, and a multistatic target tracker. Results are shown for a simple simulated scenario, showing the advantages of this adaptive ping control algorithm compared to using a preplanned, non-adaptive ping transmission schedule.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.