SDN Flow Entry Management Using Reinforcement Learning

Abstract

Modern information technology services largely depend on cloud infrastructures to provide their services. These cloud infrastructures are built on top of Datacenter Networks (DCNs) constructed with high-speed links, fast switching gear, and redundancy to offer better flexibility and resiliency. In this environment, network traffic includes long-lived (elephant) and short-lived (mice) flows with partitioned/aggregated traffic patterns. Although SDN-based approaches can efficiently allocate networking resources for such flows, the overhead due to network reconfiguration can be significant. With limited capacity of Ternary Content-Addressable Memory (TCAM) deployed in an OpenFlow enabled switch, it is crucial to determine which forwarding rules should remain in the flow table and which rules should be processed by the SDN controller in case of a table-miss on the SDN switch. This is needed in order to obtain the flow entries that satisfy the goal of reducing the long-term control plane overhead introduced between the controller and the switches. To achieve this goal, we propose a machine learning technique that utilizes two variations of Reinforcement Learning (RL) algorithms—the first of which is a traditional RL-based algorithm, while the other is deep reinforcement learning-based. Emulation results using the RL algorithm show around 60% improvement in reducing the long-term control plane overhead and around 14% improvement in the table-hit ratio compared to the Multiple Bloom Filters (MBF) method, given a fixed size flow table of 4KB.

Deep Reinforcement Learning for the Management of Software-Defined Networks and Network Function Virtualization in an Edge-IoT Architecture

“…In this sense, and following partly the methodology of Mu et al [74], based on Mnih et al [81], it is necessary to define the different states, actions and reward mechanisms that will feed the Deep Q-Networks.…”

“…As in Reference [74], we use ReLU (Rectifier Lineal Unit) functions as activation functions, as provided by Equation (11). For the experimentation a Mininet 2.2.2 virtual machine on an Ubuntu 20.4 LTS running directly on Oracle Cloud with a 1 vCPU and 1GB of RAM has been used.…”

“…One of the most relevant works in this sense is the approach by Mu et al [74], which proposes a SDN flow entry management mechanism based on Deep Q-Networks and compared with classic Q-learning reinforcement learning using Mininet emulator [75]. In our work, Deep Q-Networks is proposed to manage the Network Function Virtualization in Edge-IoT scenarios.…”

“…In our work, Deep Q-Networks is proposed to manage the Network Function Virtualization in Edge-IoT scenarios. Unlike the work of Mu et al [74], whose methodology serves as a basis for the experimentation stage of our work, our proposal is oriented towards a specific SDN management solution in a reference architecture aimed to the design, implementation and deployment of Edge-IoT solutions, in which the flows between nodes are almost entirely mice flows (that carries small amounts of data).…”

Service and Resource Aware Flow Management Scheme for an SDN-Based Smart Digital Campus Environment

“…The centralized controller installs these computed rules over the programmable forwarding devices to enable the transfer of the corresponding flows [13] [50]. Nonetheless, with the continuous expansion in the flow number, data rates, and the requirements for detailed analysis, this approach seems to have limited scalability, and it leads to long processing time and heavy overhead that affect the performance of the delay-sensitive flows [50]- [52].…”