Effective Scheduling Function Design in SDN Through Deep Reinforcement Learning

Huang, Victoria; Chen, Gang; Fu, Qiang

doi:10.1109/icc.2019.8761938

Cited by 9 publications

(7 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An RL-based algorithm uses the deep neural network (DNN) to represent its value function, called the Deep-Q (DQ) Scheduler, which provides nearly twofold performance improvement compared to the state-ofthe-art SDN controller synchronization solutions. However, some authors [29] use the RL for autonomous cyber defense in SDN and also use RL to resolve the synchronization issues of multiple controllers [30], [31]. Several AI techniques used in the SDN context have been introduced in [32], including different security and placement issues.…”

Section: Related Workmentioning

confidence: 99%

Optimizing the Lifetime of Software Defined Wireless Sensor Network via Reinforcement Learning

et al. 2021

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Optimizing the Lifetime of Software Defined Wireless Sensor Network via Reinforcement Learning

et al. 2021

View full text Add to dashboard Cite

“…Aiming at properly distributing its requests among all controllers so as to make the best use of controller capacity and achieve high network performance, every SDN switch often follows a request dispatching (RD) policy to select suitable controllers to process each newly arriving request. Clearly, carefully designing such a policy is of paramount importance to the overall functioning of multi-controller SDNs [3], [5].…”

Section: Introductionmentioning

confidence: 99%

“…Meanwhile, obtaining timely global information over the entire SDN network can cause substantial communication overhead [25]. Even though these issues can be alleviated by employing multiple co-learning agents as demonstrated in [5], the single-agent DRL algorithm cannot cope with inter-agent interference and localized network information, resulting in poor and unpredictable network performance.…”

Section: Introductionmentioning

confidence: 99%

Multi-Agent Deep Reinforcement Learning for Request Dispatching in Distributed-Controller Software-Defined Networking

Huang,

Chen,

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, distributed controller architectures have been quickly gaining popularity in Software-Defined Networking (SDN). However, the use of distributed controllers introduces a new and important Request Dispatching (RD) problem with the goal for every SDN switch to properly dispatch their requests among all controllers so as to optimize network performance. This goal can be fulfilled by designing an RD policy to guide distribution of requests at each switch. In this paper, we propose a Multi-Agent Deep Reinforcement Learning (MA-DRL) approach to automatically design RD policies with high adaptability and performance. This is achieved through a new problem formulation in the form of a Multi-Agent Markov Decision Process (MA-MDP), a new adaptive RD policy design and a new MA-DRL algorithm called MA-PPO. Extensive simulation studies show that our MA-DRL technique can effectively train RD policies to significantly outperform man-made policies, modelbased policies, as well as RD policies learned via single-agent DRL algorithms.

show abstract

“…In particular, the policy π θ takes the state s t as inputs and outputs an action a t . In our previous work [136], the action corresponds to the chosen controller for request processing. This design requires repeated processing of the policy with respect to every new request, preventing efficient use of the policy in large traffic-intensive networks.…”

Section: Policymentioning

confidence: 99%

“…The architecture BLAC proposed in Chapter 3 enables switches dispatch requests to any controllers without the switch-controller binding constraint, effectively alleviating the workload imbalance issues. With the request dispatching flexibility provided by BLAC, designing a policy to properly dispatch requests originated from every switch to suitable controllers is of paramount importance to the overall functioning of multicontroller SDNs [136,138]. Motivated by this understanding, we aim to address the Request Dispatching Policy Design (RDPD) problem in this chapter.…”

Section: Introductionmentioning

confidence: 99%

Achieving Effective Resource Management in Distributed SDN Controller Architectures

Huang¹

View full text Add to dashboard Cite

<p>As an emerging computer networking paradigm, Software-Defined Networking (SDN) empowers network operators with simplified network configuration and centralized network management. Recently, distributed controller architectures have become a notable invention where multiple controllers are jointly deployed in the network for request processing. One major research challenge for distributed controller architectures is to effectively manage the controller resources including allocating sufficient controllers to the suitable network locations and making the best use of the given controller resources. In general, existing approaches for managing the controller resources in the literature can be classified into three main directions. Designing new controller architectures belongs to the first direction, where the focus is on enabling workload shifting among controllers using switch migration. Designing controller placement algorithms to identify the number and locations of controllers is the second direction. Given the controller placement solution, the third direction is controller scheduling which aims to make the best use of the shared controllers by properly distributing requests among them. However, existing approaches have three major limitations. First, existing controller architectures feature a switch-controller binding which restricts the requests generated by a switch to only be processed by a predefined controller. Since each switch comes with different workload and the workload can be time-variant, the binding renders the bound controller susceptible to either being overloaded or underloaded. Second, existing placement algorithms have consistently underestimated the importance of controller scheduling. Due to the NP-hardness of the placement problem, Genetic Algorithm (GA) is a promising candidate. However, as a population-based approach, GA can be computationally expensive. Especially in a large network, the corresponding search space becomes too large for GA to handle effectively. Third, existing approaches for controller scheduling are mostly designed under the switch-controller binding constraint. When the scheduling is performed at a per-request level, the scheduling complexity increases significantly, rendering the efficiency and effectiveness of existing algorithms questionable. Apart from that, existing studies mainly focus on manually designing request dispatching policy which strongly relies on domain knowledge and involves a time-consuming fine-tuning process. The overall goal of this thesis is to effectively manage the controller resources in distributed SDN controller architectures. To address the three major limitations, three research objectives are established. First, this thesis aims to propose a new controller architecture to enable flexible controller placement and scheduling. Second, the thesis focuses on effectively and scalably identifying suitable controller placement while jointly taking the controller scheduling problem into consideration. Third, the thesis seeks to incorporate machine learning techniques in the request dispatching policy design to automatically learn adaptive and effective policies. To achieve the first objective, this thesis proposes a new BindingLess Architecture for distributed Controllers (BLAC) which features bindingless association between switches and controllers. With the newly introduced scheduling layer, requests can be transparently and flexibly dispatched among multiple controllers without invoking the time-consuming and complicated switch migration. Experiments conducted in this thesis show that BLAC significantly reduces the average response time and improves the throughput compared to existing SDN architectures. To achieve the second objective, this thesis proposes a Clustering-based Genetic Algorithm with Cooperative Clusters (CGA-CC) to tackle the controller placement problem. Particularly, CGA-CC partitions a large network into non-overlapping sub-networks to substantially reduce the search space of GA. Within each sub-network, GA is applied to identifying the placement solution. The quality of any given placement solution is evaluated by a gradient-descent-based scheduling algorithm which is developed to optimize the probability distribution of requests among all controllers. Moreover, a greedy load re-distribution mechanism is developed to handle unexpected demand variations by dynamically forwarding indigestible requests to adjacent sub-networks. Extensive simulations show that our algorithms can significantly outperform several existing and state-of-the-art algorithms and is more robust in handling unexpected traffic bursts. To achieve the third objective, this thesis proposes a Multi-Agent (MA) deep-reinforcement-learning-based approach with the aim to automatically learn adaptive, effective, and efficient policies used by each switch. In particular, a new adaptive policy representation is proposed to support networks with a changing number of controllers. To enable the training of an adaptive policy, a new policy gradient calculation technique is developed. Then the policy design problem is formulated as an MA Markov Decision Processing and a new MA training algorithm is proposed. The results show that the policy designed by our algorithm can easily adapt to networks with a changing number of controllers. Moreover, our policy can achieve significantly better performance compared with existing policies including the man-made policy (e.g., weighted round-robin), the model-based policy (e.g., the gradient-descent-based scheduling algorithm), and policies designed by other reinforcement learning algorithms (e.g., the proximal policy optimization algorithm).</p>

show abstract

Effective Scheduling Function Design in SDN Through Deep Reinforcement Learning

Cited by 9 publications

References 21 publications

Optimizing the Lifetime of Software Defined Wireless Sensor Network via Reinforcement Learning

Optimizing the Lifetime of Software Defined Wireless Sensor Network via Reinforcement Learning

Multi-Agent Deep Reinforcement Learning for Request Dispatching in Distributed-Controller Software-Defined Networking

Achieving Effective Resource Management in Distributed SDN Controller Architectures

Contact Info

Product

Resources

About