DeepCQ+: Robust and Scalable Routing with Multi-Agent Deep Reinforcement Learning for Highly Dynamic Networks

Kaviani, Saeed; Ryu, Bo; Ahmed, Ejaz; Larson, Kevin A.; Le, Anh N.; Yahja, Alex; Kim, Jaehwan

doi:10.1109/milcom52596.2021.9652948

Cited by 10 publications

(13 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper presents a QoS-assured intelligent routing system for WMNs with high traffic loads that use reinforcement learning to improve performance [27]. As per Saeed Kaviani et al, DeepCQ+ routing, which combines emerging multiagent deep reinforcement learning (MADRL) in a novel way, achieves persistently higher performance across a wide range of MANET architectures while training only on a limited range of network parameters and conditions [28]. In a fully decentralized environment, Xinyu You et al developed a unique packet routing framework based on multi-agent deep reinforcement learning.…”

Section: Overview Of Related Workmentioning

confidence: 99%

“…An optimal Q-table contains values that allow the AI agent to take the best action in any possible State, thus providing the agent with the potential path to the highest reward. The Qtable, therefore, represents the AI agent's policy for acting in the current environment [28]. In Q-learning, the Temporal Differences' TD' provide a method of calculating how much the Q-value for the action taken in the previous State should be changed based on what an AI agent has learned about the Q-values for the current State's actions.…”

Section: F Working Of Q-learningmentioning

confidence: 99%

See 1 more Smart Citation

Adaptive Routing in Wireless Mesh Networks Using Hybrid Reinforcement Learning Algorithm

Mahajan

Kotecha

2022

IEEE Access

View full text Add to dashboard Cite

Wireless mesh networks are popular due to their adaptability, easy-setup, flexibility, cost, and transmission time-reductions. The routing algorithm plays a vital role in transferring the data between the nodes. The network's performance is significantly impacted by the route opted by the algorithm. The router takes the decision to send the packet to the next router as per the policy of that algorithm. So even though that decision does not favor the right path selection, the router tends to follow its policy. This can be avoided by having intelligent routers that can make routing decisions on the fly.This paper presents the QL-Feed Forward routing algorithm(QFFR), a new generation of routing algorithms that combines reinforcement learning based on the Q-learning algorithm with a Feed Forward neural network. This algorithm(QFFR) can learn from the network environment and make routing decisions based on the algorithm's learnings. The AI agent's ability to select the fastest path, which enhances the efficiency of the routing operation, is demonstrated by the working of the suggested QFFR algorithm. This paper also evaluates the performance of traditional algorithms, namely, Ad-hoc On-Demand Distance-Vector, Optimized-Link-State-routing, Destination-Sequenced Distance-Vector and Distance Source routing. The evaluation parameters include throughput, packet delivery ratio, and delay. The parameters are the outcomes of the time the information takes to reach from source to destination. This analysis highlights the improvement in the routing decision ability of a router. As per analysis, Ad hoc On-Demand Distance Vector Algorithm outperforms with throughput 723.13 Kbps, delay 343.73 ns. Q-learning agent identifies the route and reaches the destination in average of 3.7s in non-grid architecture. The Q-learning agent takes 0.49sec with a grid size ten by ten and 0.53sec in three by four grid size. The suggested QFFR takes 7.62s score-over time with stable, consistent performance. INDEX TERMSDeep Learning , Reinforcement Learning, Q-learning , Markov Decision Process , routing algorithms, wireless mesh networks I. INTRODUCTION D URING this era of digital development, networks play a critical role. Figure 1 shows Wireless Mesh Networks with a mesh architecture. Wireless mesh networks' key advantages are their versatility and customizing capabilities. Any future modifications would be simple to accommodate, resulting in lower expenses and upkeep of the network. Wireless mesh networking is a recent development that emerged from a decade of Ad-hoc networking development [1]. A wireless mesh network (WMN) is a decentralized network system built on existing wireless technology, namely the 802.11 standards, that works on an Ad-hoc communication mode [2] [3]. The wireless mesh network is ideal for nextgeneration communication with its flexibility and extensive coverage. Multi-radio mesh routers and single-radio mesh clients form Wireless Mesh Networks(WMN) [4]. As a result, such networks are developed by connecting wireless access...

show abstract

Section: Overview Of Related Workmentioning

confidence: 99%

Section: F Working Of Q-learningmentioning

confidence: 99%

Adaptive Routing in Wireless Mesh Networks Using Hybrid Reinforcement Learning Algorithm

Mahajan

Kotecha

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…From node to network level, security in MANETs is important 6 . Owing to the dynamic nature of MANETs, there needs to be an acceptable level of trust 7,8 . Due to the dynamic nature and characteristics of MANETs, security on the MANET is more challenge than in typical network environments with a central controller 8,9 .…”

Section: Introductionmentioning

confidence: 99%

“…6 Owing to the dynamic nature of MANETs, there needs to be an acceptable level of trust. 7,8 Due to the dynamic nature and characteristics of MANETs, security on the MANET is more challenge…”

mentioning

confidence: 99%

Proficient link state routing in mobile ad hoc network‐based deep Q‐learning network optimized with chaotic bat swarm optimization algorithm

Rahul¹,

Kaarthick²

2022

Int J Communication

View full text Add to dashboard Cite

Mobile ad-hoc network (MANET) is a category of ad-hoc network that can be reconfigurable its network. MANETS are self-organized networks, that can use the wireless links to connect various networks via mobile nodes: but it consumes more energy and it also has routing problems. This is the major drawback of being connected with the MANET technology. Therefore, this study proposes a new protocol as deep Q-learning network optimized with chaotic bat swarm optimization algorithm (CBS)-based optimized link state routing (OLSR) (CBS-OLSR) for MANET. This protocol reduces MANET energy usage and adopts OLSR multi-point relay (MPR) technology. MANET's OLSR and the CBS algorithm utilize a similar method to locate the best optimum path from source to destination node. By embedding the new improved deep Qlearning and OLSR algorithms, both are used for optimizing the MPR sets selection, it can efficiently diminish the energy consumption in the network topology, but automatically increase the lifespan of the network. It also enhances the package delivery ratio and decreases end-to-end delay. The experimental outcomes prove that the proposed protocol is reliable and proficient that is appropriate for numerous MANET applications. K E Y W O R D Schaotic bat swarm optimization, deep Q-learning algorithm, end-to-end delay, mobile adhoc network, MPR (multi-point relay), optimized link state routing, packet delivery rate | INTRODUCTIONMANET is an autonomous system that consists of individually connected set of self-configuring nodes that is implemented using a number of techniques, viz Bluetooth or WLAN. 1 The feature of MANET is autonomous, due to every node treating its host and router simultaneously. 2 To communicate among the distant nodes, they depend on direct communication with multi-hops. These advantages allow them to accommodate multiple nodes and decentralize management with its structure at any time. 3 Unlike other wireless systems, MANET has lack of central authority to monitor traffic transmission. 4 MANET systems have an infrastructure with a lesser configuration of corresponding nodes that are connected by wireless connections. The nodes are moving spontaneously or rather in a random section. 5 From node to network level, security in MANETs is important. 6 Owing to the dynamic nature of MANETs, there needs to be an acceptable level of trust. 7,8 Due to the dynamic nature and characteristics of MANETs, security on the MANET is more challenge

show abstract

“…Researchers have applied DRL algorithms for various problems in mobile ad-hoc networks (MANETs), e.g., for minimizing average or worst-case end-to-end delay in routing problems [2], [3] and routing path optimization [4]. In [5], it is shown that the DRL-based DeepCQ+ algorithm outperforms the state-of-the-art robust routing for dynamic networks (R2DN) [6]. In this context, robust routing relies on the routing of the unicast data flows with the availability of simplified multicast forwarding (SMF) (as simplified data flooding and relaying) among routing peers [7].…”

Section: Introductionmentioning

confidence: 99%

DeepADMR: A Deep Learning based Anomaly Detection for MANET Routing

Yahja

Kaviani

Ryu

et al. 2022

MILCOM 2022 - 2022 IEEE Military Communications Conference (MILCOM)

Self Cite

View full text Add to dashboard Cite

We developed DeepADMR, a novel neural anomaly detector for the deep reinforcement learning (DRL)-based DeepCQ+ MANET routing policy. The performance of DRLbased algorithms such as DeepCQ+ is only verified within the trained and tested environments, hence their deployment in the tactical domain induces high risks. DeepADMR monitors unexpected behavior of the DeepCQ+ policy based on the temporal difference errors (TD-errors) in real-time and detects anomaly scenarios with empirical and non-parametric cumulative-sum statistics. The DeepCQ+ design via multi-agent weight-sharing proximal policy optimization (PPO) is slightly modified to enable the real-time estimation of the TD-errors. We report the DeepADMR performance in the presence of channel disruptions, high mobility levels, and network sizes beyond the training environments, which shows its effectiveness.

show abstract

DeepCQ+: Robust and Scalable Routing with Multi-Agent Deep Reinforcement Learning for Highly Dynamic Networks

Cited by 10 publications

References 26 publications

Adaptive Routing in Wireless Mesh Networks Using Hybrid Reinforcement Learning Algorithm

Adaptive Routing in Wireless Mesh Networks Using Hybrid Reinforcement Learning Algorithm

Proficient link state routing in mobile ad hoc network‐based deep Q‐learning network optimized with chaotic bat swarm optimization algorithm

DeepADMR: A Deep Learning based Anomaly Detection for MANET Routing

Contact Info

Product

Resources

About