Deep reinforcement learning meets graph neural networks: Exploring a routing optimization use case

Almasan, Paul; Suárez-Varela, José; Badia-Sampera, Arnau; Rusek, Krzysztof; Barlet‐Ros, Pere; Cabellos‐Aparicio, Albert

doi:10.1016/j.comcom.2022.09.029

Cited by 72 publications

(25 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [23], a problem-specific action space is designed using Deep Reinforcement Learning (Deep RL) agents and GNNs to enable generalization. The proposed GNN-based DRL agent is capable of learning and generalizing over arbitrary network topologies.…”

Section: Related Workmentioning

confidence: 99%

Experiences With Deep Learning Enhanced Steering Mechanisms for Debugging of Fundamental Cloud Services

2023

View full text Add to dashboard Cite

Cloud architecture blueprints or reference architectures allow the reuse of existing knowledge and best practices when creating new cloud native solutions. Therefore, debugging reference architecture candidates (or their new versions) is an extremely crucial but tedious and time-consuming task due to the deployment of complex services in typical multi-tenant and non-deterministic environments. During the debugging/testing/maintenance scenarios, we might be able to achieve greater levels of test coverage (and eventually improved reliability) by modelling and verifying at least their most fundamental building blocks. The main objective of our work is to integrate the stochastic modelling and verification techniques based on deep learning methods into the debugging cycle in order to handle large state spaces more efficiently, i.e. by steering the process of traversing state space towards suspicious situations that may result in potential bugs in the actual system with smart steering during the traversal. For this purpose, our presented and illustrated approach combines (among others) Continuous Time Markov Chain modelling (CTMC) techniques with deep learning methods including autoencoder, Long Short-Term Memory (LSTM) and Graph Neural Network (GNN) models. Our experiences are summarized with widespread cloud design patterns including load balancing and service mesh topologies.

show abstract

Section: Related Workmentioning

confidence: 99%

Experiences With Deep Learning Enhanced Steering Mechanisms for Debugging of Fundamental Cloud Services

2023

View full text Add to dashboard Cite

show abstract

“…Ou et al [146] address recurrent disease prevention on a social network by using GCNs and a two-level hierarchical DQN framework. Other applications can be seen in Virtual Network Function (VNF) placement [166], adversarial attacks [168], connected autonomous vehicles [30], VNF forward graph placement [204], route optimization [6], and mobile crowdsourcing [205]. They are all cases of successful application of ML method on graph-based problems.…”

Section: Graph Autoencodersmentioning

confidence: 99%

A Survey on Influence Maximization: From an ML-Based Combinatorial Optimization

Li¹,

Gao²,

Yunxuan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Influence Maximization (IM) is a classical combinatorial optimization problem, which can be widely used in mobile networks, social computing, and recommendation systems. It aims at selecting a small number of users such that maximizing the influence spread across the online social network. Because of its potential commercial and academic value, there are a lot of researchers focusing on studying the IM problem from different perspectives. The main challenge comes from the NP-hardness of the IM problem and #P-hardness of estimating the influence spread, thus traditional algorithms for overcoming them can be categorized into two classes: heuristic algorithms and approximation algorithms. However, there is no theoretical guarantee for heuristic algorithms, and the theoretical design is close to the limit. Therefore, it is almost impossible to further optimize and improve their performance.With the rapid development of artificial intelligence, the technology based on Machine Learning (ML) has achieved remarkable achievements in many fields. In view of this, in recent years, a number of new methods have emerged to solve combinatorial optimization problems by using ML-based techniques. These methods have the advantages of fast solving speed and strong generalization ability to unknown graphs, which provide a brand-new direction for solving combinatorial optimization problems. Therefore, we abandon the traditional algorithms based on iterative search and review the recent development of ML-based methods, especially Deep Reinforcement Learning, to solve the IM problem and other variants in social networks. We focus on summarizing the relevant background knowledge, basic principles, common methods, and applied research. Finally, the challenges that need to be solved urgently in future IM research are pointed out. CCS Concepts: • Networks → Network algorithms; • Theory of computation → Design and analysis of algorithms; • Computing methodologies → Machine learning approaches; Knowledge representation and reasoning.

show abstract

“…The authors in [ 27 ] proposed an intelligent routing algorithm combining the graph neural network (GNN) and deep deterministic policy gradient (DDPG) in the SDN environment, which can be effectively extended to different network topologies, improving load-balancing capabilities and generalizability. The authors in [ 28 ] combined GNN with the DQN algorithm to address the lack of generalization abilities in untrained OTN topologies. OTN topology graphs are non-Euclidean data, and the nodes in their topology graphs typically contain useful feature information that most neural networks are unable to comprehend.…”

Section: Related Researchmentioning

confidence: 99%

A Routing Optimization Method for Software-Defined Optical Transport Networks Based on Ensembles and Reinforcement Learning

Chen

Xiao

et al. 2022

Sensors

View full text Add to dashboard Cite

Optical transport networks (OTNs) are widely used in backbone- and metro-area transmission networks to increase network transmission capacity. In the OTN, it is particularly crucial to rationally allocate routes and maximize network capacities. By employing deep reinforcement learning (DRL)- and software-defined networking (SDN)-based solutions, the capacity of optical networks can be effectively increased. However, because most DRL-based routing optimization methods have low sample usage and difficulty in coping with sudden network connectivity changes, converging in software-defined OTN scenarios is challenging. Additionally, the generalization ability of these methods is weak. This paper proposes an ensembles- and message-passing neural-network-based Deep Q-Network (EMDQN) method for optical network routing optimization to address this problem. To effectively explore the environment and improve agent performance, the multiple EMDQN agents select actions based on the highest upper-confidence bounds. Furthermore, the EMDQN agent captures the network’s spatial feature information using a message passing neural network (MPNN)-based DRL policy network, which enables the DRL agent to have generalization capability. The experimental results show that the EMDQN algorithm proposed in this paper performs better in terms of convergence. EMDQN effectively improves the throughput rate and link utilization of optical networks and has better generalization capabilities.

show abstract

Deep reinforcement learning meets graph neural networks: Exploring a routing optimization use case

Cited by 72 publications

References 30 publications

Experiences With Deep Learning Enhanced Steering Mechanisms for Debugging of Fundamental Cloud Services

Experiences With Deep Learning Enhanced Steering Mechanisms for Debugging of Fundamental Cloud Services

A Survey on Influence Maximization: From an ML-Based Combinatorial Optimization

A Routing Optimization Method for Software-Defined Optical Transport Networks Based on Ensembles and Reinforcement Learning

Contact Info

Product

Resources

About