Effective utilization of mobile ad hoc underwater distributed networks is challenging due to high system costs and the harsh environment characterized by low bandwidth, large latency, high energy consumption, and node mobility. This work addresses the routing issue, which is critical in successfully establishing and utilizing an underwater network. In particular, it focuses on reinforcement learning (RL)-based routing algorithms, which possess the ability to explore the network environment and adapt routing decisions to the constantly changing topology of the network due to node mobility and energy usage. This paper presents a routing algorithm based on Q-learning, one of the RL approaches, with additional Kinematic and Sweeping features, therefore referred to as QKS. These two additional features are introduced to address the potential slow convergence associated with pure RL algorithms. The results of a detailed packet-level simulation have been obtained using the NS-2 open-source network simulator with underwater modeling additions. The energy efficiency, convergence, and delivery performance of QKS are compared with two other routing protocols for underwater networks, a basic flooding approach (ICRP (Liang, 2007)) and a basic Q-learning implementation (QELAR (Hu, 2010)), using simulations of networks with both fixed and mobile nodes.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.