TIDE: Time-relevant deep reinforcement learning for routing optimization

Sun, Peng; Hu, Yuxiang; Lan, Julong; Tian, Le; Chen, Min

doi:10.1016/j.future.2019.04.014

Cited by 84 publications

(33 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1b. Prior studies with indirect routing representations, e.g., through link weights, have typically only considered single-path routing [19], [20], see Fig. 1a.…”

Section: B Contribution: Direct Flow Routing Representation For Flowmentioning

confidence: 99%

“…The so-called "sample inefficiency" of deep reinforcement learning [9], [10] (in contrast to the classical tabular Q-Learning [7]) leads to very long training times for the deep neural network, resulting in potentially excessive solution computation times. For instance, the recent TIDE study [19] considered around one hundred training episodes, each with one thousand time steps, whereby each time step is on the order of one second, resulting in training times on the order of days. Novel computation strategies can reduce the time complexity with appropriate initialization [24], [25], however timely deep reinforcement learning continues to be a challenge.…”

Section: B Contribution: Direct Flow Routing Representation For Flowmentioning

confidence: 99%

See 1 more Smart Citation

QR-SDN: Towards Reinforcement Learning States, Actions, and Rewards for Direct Flow Routing in Software-Defined Networks

et al. 2020

View full text Add to dashboard Cite

Flow routing can achieve fine-grained network performance optimizations by routing distinct packet traffic flows over different network paths. While the centralized control of Software-Defined Networking (SDN) provides a control framework for implementing centralized network optimizations, e.g., optimized flow routing, the implementation of flow routing that is adaptive to varying traffic loads requires complex models. The goal of this study is to pursue a model-free approach that is based on reinforcement learning. We design and evaluate QR-SDN, a classical tabular reinforcement learning approach that directly represents the routing paths of individual flows in its state-action space. Due to the direct representation of flow routes in the QR-SDN state-action space, QR-SDN is the first reinforcement learning SDN routing approach to enable multiple routing paths between a given source (ingress) switch-destination (egress) switch pair while preserving the flow integrity. That is, in QR-SDN, packets of a given flow take the same routing path, while different flows with the same source-destination switch pair may take different routes (in contrast, the recent DRL-TE approach splits a given flow on a per-packet basis incurring high complexity and out-of-order packets). We implemented QR-SDN in a Software-Defined Network (SDN) emulation testbed. Our evaluations demonstrate that the flow-preserving multi-path routing of QR-SDN achieves substantially lower flow latencies than prior routing approaches that determine only a single source-destination route. A limitation of QR-SDN is that the state-action space grows exponentially with the number of network nodes. Addressing the scalability of direct flow routing, e.g., through routing only high-rate flows, is an important direction for future research. The QR-SDN code is made publicly available to support this future research.

show abstract

“…1b. Prior studies with indirect routing representations, e.g., through link weights, have typically only considered single-path routing [19], [20], see Fig. 1a.…”

Section: B Contribution: Direct Flow Routing Representation For Flowmentioning

confidence: 99%

Section: B Contribution: Direct Flow Routing Representation For Flowmentioning

confidence: 99%

QR-SDN: Towards Reinforcement Learning States, Actions, and Rewards for Direct Flow Routing in Software-Defined Networks

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Mestres et al [112] Sun et al [114] proposed an intelligent network control architectural framework that employs DRL to dynamically optimize routing plans in an SDN-enabled network without the need for human involvement. The proposed framework in called TIDE.…”

Section: B ML and Dl Techniques For Routing Optimization In Sdnmentioning

confidence: 99%

“…Based on the supervised ML module, LSTM-RNN algorithm is employed to extract shortterm network data traffic variabilities and periodicities, resulting in the meaningful features which are combined at the integration step to ensure traffic flow prediction and energy-efficient routing with guaranteed QoS performance. On the other hand, the DRL module performs learning from the existing historical data or right from scratch by iteratively interfacing with the defined network setting [114][115][116][117][118] [130]. Using publicly available dataset, the module can be evaluated in terms of accuracy and convergence speed.…”

Section: B Description Of the Supervised ML And Drl Frameworkmentioning

confidence: 99%

AI-Assisted Framework for Green-Routing and Load Balancing in Hybrid Software-Defined Networking: Proposal, Challenges and Future Perspective

et al. 2020

View full text Add to dashboard Cite

The explosive growth of IP networks, the advent of cloud computing, and the rapid progress in wireless communications witnessed today reflect significant progress towards meeting the explosive data traffic demands. Consequently, communications service providers should deploy efficient and intelligent network solutions to accommodate the huge traffic demands and to ease the capacity pressure on their network infrastructure. Besides, vendors should develop novel energy-efficient networks to reduce network utility costs and carbon footprint. Software-defined networking (SDN) provides a suitable solution, however, complete SDN deployment is currently unachievable in the short-term. An alternative is the hybrid SDN/ open shortest path forwarding (OSPF) network, which allows the deployment of SDN in legacy networks. Nevertheless, hybrid SDN/OSPF also faces several technical, economic and organizational challenges. Although many energyefficiency routing solutions exist in hybrid SDN/OSPF networks, they are generic and reactive by design. Moreover, these solutions are characterized by manual control plane forwarding configurations, leading to sub-optimal performance. The recent promising combination of SDN and artificial intelligence (AI) techniques such as machine learning (ML) and deep learning (DL) in traffic management and control provides tremendous opportunities. In this paper, we first provide a review of the most recent optimization approaches for global energy-efficient routing and load balancing. Next, we investigate a scalable and intelligent integrated architectural framework that leverages deep reinforcement learning (DRL) techniques to realize predictive and rate adaptive energy-efficient routing with guaranteed quality of service (QoS), in transitional hybrid SDN/OSPF networks. Based on the need to minimize global network energy consumption and improve link performance, this paper provides key research insights into the current progress in hybrid SDN/OSPF, ML and AI in the hope of stimulating more research.

show abstract

“…With the explosive growth of next-generation Internet of Things (IoT) applications, the delay requirements of users for content are becoming increasingly higher, such as virtual reality application [1], [2]. However, due to the distance between the content server and users, high communication delay is caused, which can not meet the user's delay requirements [3]. Fortunately, with the introduction of edge cloud, content can be cached on edge cloud to satisfy the user's quality of service (QoE) [4], [5].…”

Section: Introductionmentioning

confidence: 99%

Human-Like Hybrid Caching in Software-Defined Edge Cloud

Hao

et al. 2020

IEEE Internet Things J.

Self Cite

View full text Add to dashboard Cite

With the development of Internet of Things (IoT) and communication technology, the number of next-generation IoT devices has increased explosively, and the delay requirement for content requests is becoming progressively higher. Fortunately, the edge-caching scheme can satisfy users' demands for low latency of content. However, the existing caching schemes are not smart enough. To address these challenges, we propose a human-like hybrid caching architecture based on the software defined edge cloud, which simultaneously considers the content popularity and the fine-grained user characteristics. Then, an optimization problem with a caching hit ratio as an optimization objective is formulated. To solve this problem, using reinforcement learning, we design a human-like hybrid caching algorithm. Extensive experiments show that compared with popular caching schemes, human-like hybrid caching schemes can improve the cache hit ratio by 20%.

show abstract

TIDE: Time-relevant deep reinforcement learning for routing optimization

Cited by 84 publications

References 10 publications

QR-SDN: Towards Reinforcement Learning States, Actions, and Rewards for Direct Flow Routing in Software-Defined Networks

QR-SDN: Towards Reinforcement Learning States, Actions, and Rewards for Direct Flow Routing in Software-Defined Networks

AI-Assisted Framework for Green-Routing and Load Balancing in Hybrid Software-Defined Networking: Proposal, Challenges and Future Perspective

Human-Like Hybrid Caching in Software-Defined Edge Cloud

Contact Info

Product

Resources

About