Deep Q-Learning for Routing Schemes in SDN-Based Data Center Networks

Fu, Qiongxiao; Sun, Enchang; Meng, Ke; Li, Meng; Zhang, Yanhua

doi:10.1109/access.2020.2995511

Cited by 54 publications

(21 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…To the best of our knowledge, our study is the first to directly represent flow routing in the stateaction space of reinforcement learning. For completeness we note that deep reinforcement learning has been applied to a wide variety of other communication network problems, including distributed routing [58], [59], congestion control [60], data center networks [61], wireless network routing [62]- [71], vehicular ad hoc network routing [72], [73], optical networking [74]- [76], caching [77], and mobile edge computing [78], [79]. We also note that a preprocessing approach for efficiently representing virtual network embeddings for subsequent algorithm processing has been examined in [80].…”

Section: Review Of Related Workmentioning

confidence: 99%

QR-SDN: Towards Reinforcement Learning States, Actions, and Rewards for Direct Flow Routing in Software-Defined Networks

et al. 2020

View full text Add to dashboard Cite

Flow routing can achieve fine-grained network performance optimizations by routing distinct packet traffic flows over different network paths. While the centralized control of Software-Defined Networking (SDN) provides a control framework for implementing centralized network optimizations, e.g., optimized flow routing, the implementation of flow routing that is adaptive to varying traffic loads requires complex models. The goal of this study is to pursue a model-free approach that is based on reinforcement learning. We design and evaluate QR-SDN, a classical tabular reinforcement learning approach that directly represents the routing paths of individual flows in its state-action space. Due to the direct representation of flow routes in the QR-SDN state-action space, QR-SDN is the first reinforcement learning SDN routing approach to enable multiple routing paths between a given source (ingress) switch-destination (egress) switch pair while preserving the flow integrity. That is, in QR-SDN, packets of a given flow take the same routing path, while different flows with the same source-destination switch pair may take different routes (in contrast, the recent DRL-TE approach splits a given flow on a per-packet basis incurring high complexity and out-of-order packets). We implemented QR-SDN in a Software-Defined Network (SDN) emulation testbed. Our evaluations demonstrate that the flow-preserving multi-path routing of QR-SDN achieves substantially lower flow latencies than prior routing approaches that determine only a single source-destination route. A limitation of QR-SDN is that the state-action space grows exponentially with the number of network nodes. Addressing the scalability of direct flow routing, e.g., through routing only high-rate flows, is an important direction for future research. The QR-SDN code is made publicly available to support this future research.

show abstract

Section: Review Of Related Workmentioning

confidence: 99%

QR-SDN: Towards Reinforcement Learning States, Actions, and Rewards for Direct Flow Routing in Software-Defined Networks

et al. 2020

View full text Add to dashboard Cite

show abstract

“…al. [38] presented a routing strategy based on deep Qlearning (DQL) to generate optimal routing paths autonomously for SDN-based data center networks. However, they aimed to provide different quality of service guarantees for mice-flows and elephant-flows, designated in a data center network.…”

Section: Related Workmentioning

confidence: 99%

“…In the control plane, the well-established Ryu controller [39] is adopted. There are three modules in the controller, topology discovery, period monitor, and an ANN model, which are used to explore In recent years, several routing mechanisms [32][33][34][35][36][37][38] combined with AI technology have been proposed to enhance learning ability from past experiences and smart route-decision capability, thereby improving overall network performance. First of all, the works in [33][34][35] introduced AI technology to routing protocols for wireless sensor networks in order to improve the energy consumption of each node.…”

Section: System Architecturementioning

confidence: 99%

“…The main contribution of AMPS controller in comparison to SDN with traditional routing is its ensuring high availability of an unloaded path for high priority flows even in a heavily loaded network. Fu et al [38] presented a routing strategy based on deep Q-learning (DQL) to generate optimal routing paths autonomously for SDN-based data center networks. However, they aimed to provide different quality of service guarantees for mice-flows and elephant-flows, designated in a data center network.…”

Section: System Architecturementioning

confidence: 99%

See 1 more Smart Citation

Artificial Intelligence Enabled Routing in Software Defined Networking

Hwang²,

Hwang³

et al. 2020

Applied Sciences

View full text Add to dashboard Cite

Software defined networking (SDN) is an emerging networking architecture that separates the control plane from the data plane and moves network management to a central point, called the controller. The controller is responsible for preparing the flow tables of each switch in the data plane. Although dynamic routing can perform rerouting in case of congestion by periodically monitoring the status of each data flow, problems concerning a suitable monitoring period duration and lack of learning ability from past experiences to avoid similar but ineffective route decisions remain unsolved. This paper presents an artificial intelligence enabled routing (AIER) mechanism with congestion avoidance in SDN, which can not only alleviate the impact of monitoring periods with dynamic routing, but also provide learning ability and superior route decisions by introducing artificial intelligence (AI) technology. We evaluate the performance of the proposed AIER mechanism on the Mininet simulator by installing three additional modules, namely, topology discovery, monitoring period, and an artificial neural network, in the control plane. The effectiveness and superiority of our proposed AIER mechanism are demonstrated by performance metrics, including average throughput, packet loss ratio, and packet delay versus data rate for different monitoring periods in the system.

show abstract

“…Also, the use of RL has been reported in application areas such as energy-efficient routing in SDN-enabled networks. Other studies have considered the use of RL in addressing routing with QoSguarantees [122] [131]. However, such studies are based on traditional table-based agents which cannot support the need for efficient solutions in the condition of unseen network topology state [109].…”

Section: ) Analysis Of Selected Load Balancing and Energy-efficient mentioning

confidence: 99%

AI-Assisted Framework for Green-Routing and Load Balancing in Hybrid Software-Defined Networking: Proposal, Challenges and Future Perspective

et al. 2020

View full text Add to dashboard Cite

The explosive growth of IP networks, the advent of cloud computing, and the rapid progress in wireless communications witnessed today reflect significant progress towards meeting the explosive data traffic demands. Consequently, communications service providers should deploy efficient and intelligent network solutions to accommodate the huge traffic demands and to ease the capacity pressure on their network infrastructure. Besides, vendors should develop novel energy-efficient networks to reduce network utility costs and carbon footprint. Software-defined networking (SDN) provides a suitable solution, however, complete SDN deployment is currently unachievable in the short-term. An alternative is the hybrid SDN/ open shortest path forwarding (OSPF) network, which allows the deployment of SDN in legacy networks. Nevertheless, hybrid SDN/OSPF also faces several technical, economic and organizational challenges. Although many energyefficiency routing solutions exist in hybrid SDN/OSPF networks, they are generic and reactive by design. Moreover, these solutions are characterized by manual control plane forwarding configurations, leading to sub-optimal performance. The recent promising combination of SDN and artificial intelligence (AI) techniques such as machine learning (ML) and deep learning (DL) in traffic management and control provides tremendous opportunities. In this paper, we first provide a review of the most recent optimization approaches for global energy-efficient routing and load balancing. Next, we investigate a scalable and intelligent integrated architectural framework that leverages deep reinforcement learning (DRL) techniques to realize predictive and rate adaptive energy-efficient routing with guaranteed quality of service (QoS), in transitional hybrid SDN/OSPF networks. Based on the need to minimize global network energy consumption and improve link performance, this paper provides key research insights into the current progress in hybrid SDN/OSPF, ML and AI in the hope of stimulating more research.

show abstract

Deep Q-Learning for Routing Schemes in SDN-Based Data Center Networks

Cited by 54 publications

References 19 publications

QR-SDN: Towards Reinforcement Learning States, Actions, and Rewards for Direct Flow Routing in Software-Defined Networks

QR-SDN: Towards Reinforcement Learning States, Actions, and Rewards for Direct Flow Routing in Software-Defined Networks

Artificial Intelligence Enabled Routing in Software Defined Networking

AI-Assisted Framework for Green-Routing and Load Balancing in Hybrid Software-Defined Networking: Proposal, Challenges and Future Perspective

Contact Info

Product

Resources

About