Self-Driving Network and Service Coordination Using Deep Reinforcement Learning

Abstract

Modern services comprise interconnected components, e.g., microservices in a service mesh, that can scale and run on multiple nodes across the network on demand. To process incoming traffic, service components have to be instantiated and traffic assigned to these instances, taking capacities and changing demands into account. This challenge is usually solved with custom approaches designed by experts. While this typically works well for the considered scenario, the models often rely on unrealistic assumptions or on knowledge that is not available in practice (e.g., a priori knowledge).We propose a novel deep reinforcement learning approach that learns how to best coordinate services and is geared towards realistic assumptions. It interacts with the network and relies on available, possibly delayed monitoring information. Rather than defining a complex model or an algorithm how to achieve an objective, our model-free approach adapts to various objectives and traffic patterns. An agent is trained offline without expert knowledge and then applied online with minimal overhead. Compared to a state-of-the-art heuristic, it significantly improves flow throughput and overall network utility on real-world network topologies and traffic traces. It also learns to optimize different objectives, generalizes to scenarios with unseen, stochastic traffic patterns, and scales to large real-world networks.

The State of the Art of Emergent Software Systems

“…Response time and responsiveness Filho et al [8], [9], [12], [18], [19] Response time and responsiveness Cardozo [4] X Wang et al [20]- [22] Responsiveness, availability, throughput, successability, reliability Younes [6] Hybrid (user's feedback) Mannava and Ramesh [23] Process CPU time, heap memory Ding et al [24] X Rosa et al [25] X Tan et al [26] Dependability and responsiveness Gonçalves et al [27] Responsiveness Yan et al [28] Responsiveness Belhaj et al [29] Availability, responsiveness, service calls Schneider et al [30], [31] Throughput, energy costs, efficiency Ganguly and Sakib [32] Failure rate, responsiveness Deshpande et al [33] Response time, availability, throughput, successability Kulkarni et al [34] Response time, model confidence, and CPU consumption Rainford et al [35] Response time, Resource utilization Silva et al [36] Resource utilization…”

“…As seen in Table 3, most ESS approaches are based on nonfunctional adaptation goals, including response time [8], [9], [12], [15]- [19], [33], [35], responsiveness [20]- [22], [27]- [29], availability [20]- [22], [29], throughput [20]- [22], [30], [31], [33], successability [20]- [22], [33], reliability [20]- [22], process CPU time [23], [34], heap memory [23], dependability [26], service calls [29], energy cost [30], [31] and failure rate [32]. Some approaches only target one adaptation goal at a time such as [27], [28], while others tackle multiple adaptation goals.…”

“…The goal is to allow software to evaluate its status based on the current environment conditions, thus enabling the implementation of self-adaptation strategies that optimize its adaptation goals (e.g., minimizing the response time). Table 4 outlines different monitoring approaches used by existing ESSs, including the proxy approach [12], [15], [29], the sensor factory pattern [38], the observer design pattern [38], and the Prometheus framework [30], [31].…”

Towards a Self-Driving Management System for the Automated Realization of Intents

“…In this context [90] surveys the use of unsupervised learning for next-generation networks. Proposals focused on learning include: deep reinforcement learning to provision and coordinate VNF services [91]; reinforcement learning to optimize service provisioning policies for the optical domain [92]; [93] uses a measurelearn-decide-action control loop within data and control planes for local control, and a management plane to revise globally; [94] explores the adaptability in SDN, NFV through ML-enhanced observation, composition, and control; and, [95] studies fault detection for IoT networks.…”

Self-Learning Multi-Objective Service Coordination Using Deep Reinforcement Learning

“…This paper is an extension of our previous work [15]. In this extended version, we consider QoS requirements in terms of acceptable end-to-end delay (soft and hard deadlines) as well as limited link capacities in addition to nodes' compute capacities, which further restrict the solution space and make service coordination more challenging.…”