Scanflow-K8s: Agent-based Framework for Autonomic Management and Supervision of ML Workflows in Kubernetes Clusters

Abstract

Machine Learning (ML) projects are currently heavily based on workflows composed of some reproducible steps and executed as containerized pipelines to build or deploy ML models efficiently because of the flexibility, portability, and fast delivery they provide to the ML life-cycle. However, deployed models need to be watched and constantly managed, supervised, and debugged to guarantee their availability, validity, and robustness in unexpected situations. Therefore, containerized ML workflows would benefit from leveraging flexible and diverse autonomic capabilities. This work presents an architecture for autonomic ML workflows with abilities for multi-layered control, based on an agent-based approach that enables autonomic management and supervision of ML workflows at the application layer and the infrastructure layer (by collaborating with the orchestrator). We redesign the Scanflow ML framework to support such multiagent approach by using triggers, primitives, and strategies. We also implement a practical platform, so-called Scanflow-K8s, that enables autonomic ML workflows on Kubernetes clusters based on the Scanflow agents. MNIST image classification and MLPerf ImageNet classification benchmarks are used as case studies to show the capabilities of Scanflow-K8s under different scenarios. The experimental results demonstrate the feasibility and effectiveness of our proposed agent approach and the Scanflow-K8s platform for the autonomic management of ML workflows in Kubernetes clusters at multiple layers.

Fine-Grained Scheduling for Containerized HPC Workloads in Kubernetes Clusters

“…Our fine-grained scheduling approach for containerized HPC workloads is built over the existing Scanflow-Kubernetes platform [24] [25]. It is implemented both within a Scanflow(MPI) extension package in the application layer (see in Scanflow-Kubernetes github repository 1 ) and an enhanced Volcano scheduler/controller manager in the infrastructure layer (see in Volcano github repository 2 ).…”

The Flux Operator

“…Given this landscape, any entity involved in the business of scaled computing will fall behind if these technological needs are not prioritized. 4 In cloud computing communities, machine learning workloads are also becoming increasingly important, [7][8][9][10] and the cloud container orchestration technology Kubernetes is becoming the de facto standard for orchestration of these workflows following its success orchestrating microservices. As of June of 2023, the Kubernetes project had over 74,000 contributors, making it the second largest open source project ever after Linux, and the "most widely used container orchestration platform in existence" (the CNCF project report).…”

Convergence of high performance computing, big data, and machine learning applications on containerized infrastructures

“…ML applications focus on learning models from data and making predictions by using the trained model. From a runtime perspective, the ML training and batch ML inference could be executed as offline jobs and may take days to complete, whereas the online ML inference service is realized as a long-run service that is able to deal with dynamic prediction queries from end-users [97][98] [203][92]. These bring challenges in the platform layer for containers to consider and support the development, testing, deployment, and operation of wide range types of applications.…”

“…Our contributions to achieve this objective are as follows, and have resulted in publications [99][101] [97]. BD applications have also been containerized in this manner, as published formerly by our group [160], hence, this is not included as a contribution in this thesis:…”

“…Our contributions to achieve this objective are as follows, and have resulted in publications [98][100]:…”