A Software-Defined Directional Q-Learning Grid-Based Routing Platform and Its Two-Hop Trajectory-Based Routing Algorithm for Vehicular Ad Hoc Networks

Abstract

Dealing with the packet-routing problem is challenging in the V2X (Vehicle-to-Everything) network environment, where it suffers from the high mobility of vehicles and varied vehicle density at different times. Many related studies have been proposed to apply artificial intelligence models, such as Q-learning, which is a well-known reinforcement learning model, to analyze the historical trajectory data of vehicles and to further design an efficient packet-routing algorithm for V2X. In order to reduce the number of Q-tables generated by Q-learning, grid-based routing algorithms such as the QGrid have been proposed accordingly to divide the entire network environment into equal grids. This paper focuses on improving the defects of these grid-based routing algorithms, which only consider the vehicle density of each grid in Q-learning. Hence, we propose a Software-Defined Directional QGrid (SD-QGrid) routing platform in this paper. By deploying an SDN Control Node (CN) to perform centralized control for V2X, the SD-QGrid considers the directionality from the source to the destination, real-time positions and historical trajectory records between the adjacent grids of all vehicles. The SD-QGrid further proposes the flows of the offline Q-learning training process and the online routing decision process. The two-hop trajectory-based routing (THTR) algorithm, which depends on the source–destination directionality and the movement direction of the vehicle for the next two grids, is proposed as a vehicle node to forward its packets to the best next-hop neighbor node in real time. Finally, we use the real vehicle trajectory data of Taipei City to conduct extensive simulation experiments with respect to four transmission parameters. The simulation results prove that the SD-QGrid achieved an over 10% improvement in the average packet delivery ratio and an over 25% reduction in the average end-to-end delay at the cost of less than 2% in average overhead, compared with two well-known Q-learning grid-based routing algorithms.

Stochastic Modeling for Intelligent Software-Defined Vehicular Networks: A Survey

“…To address this challenge, researchers have proposed several extensions of the Qlearning algorithm, such as the Deep Q-Network (DQN) algorithm, which uses deep neural networks to estimate the state-action value function [157]. The DQN algorithm has been shown to be effective in learning the optimal policy for complex and dynamic network environments, such as in the case of SDVNs.…”

Multi-Objective Message Routing in Electric and Flying Vehicles Using a Genetics Algorithm

“…On the other hand, it is always possible to consider the original hop count-based routing scheme which not only ensures the minimum number of hops in a route but also results in a faster route selection [ 9 ]. Nevertheless, several existing solutions have tried utilizing SDN or a multi-objective routing for ground vehicles [ 10 , 11 , 12 ]. Moreover, routing in UAVs which considers metrics such as charging stations, VANET supporting networks, and trajectory-based systems has also been proposed [ 13 , 14 , 15 , 16 , 17 , 18 , 19 ].…”