Scanflow: A multi-graph framework for Machine Learning workflow management, supervision, and debugging

Bravo-Rocca, Gusseppe; Liu, Peini; Guitart, Jordi; Dholakia, Ajay; Ellison, David; Falkanger, Jeffrey; Hodak, Miroslav

doi:10.1016/j.eswa.2022.117232

Cited by 9 publications

(13 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Traditionally, interaction has been modelled through agent communication languages, such as FIPA-ACL 13 . Recently, researchers have proposed other interaction methods based on concepts like using a shared volume [19]. Scanflow leverages microservice-based agents [23] which could also interact with each other transparently with a service discovery through RESTful APIs.…”

Section: B Agent Social Abilitymentioning

confidence: 99%

“…In this experiment, we show how the various teams will use Scanflow-K8s in the different phases to build and deploy their workflows, as well as the effectiveness of agents that help to manage and supervise the workflows at the application layer while running in production (i.e., to detect and handle drift anomalies). The complete use case is available at Github 19 .…”

Section: B Mnist Classificationmentioning

confidence: 99%

“…The Data Engineer team can provide custom functions to enhance the capabilities of each agent. This section evaluates agents which feature a non-trivial data drift detector workflow built from the implementation of the components presented in our previous paper [19]. Checker-agent detects out-of-distribution samples by means of a convolutional deep autoencoder and selects the critical points within these data, which are labelled based on human intervention.…”

Section: B Mnist Classificationmentioning

confidence: 99%

“…In our previous work, we presented Scanflow, an executor multi-graph framework for end-to-end ML workflow management and debugging in an offline mode, in the form of a proof-of-concept prototype running in a single node, which featured an anomaly detector of out-of-distribution samples in the inference phase [19]. In this paper, we contribute Scanflow-K8s, a functional agent-based MLOps framework that enables autonomic management and online supervision of the endto-end life-cycle of ML workflows on Kubernetes.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Scanflow-K8s: Agent-based Framework for Autonomic Management and Supervision of ML Workflows in Kubernetes Clusters

Liu

Bravo-Rocca

Guitart

et al. 2022

2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid)

Self Cite

View full text Add to dashboard Cite

Machine Learning (ML) projects are currently heavily based on workflows composed of some reproducible steps and executed as containerized pipelines to build or deploy ML models efficiently because of the flexibility, portability, and fast delivery they provide to the ML life-cycle. However, deployed models need to be watched and constantly managed, supervised, and debugged to guarantee their availability, validity, and robustness in unexpected situations. Therefore, containerized ML workflows would benefit from leveraging flexible and diverse autonomic capabilities. This work presents an architecture for autonomic ML workflows with abilities for multi-layered control, based on an agent-based approach that enables autonomic management and supervision of ML workflows at the application layer and the infrastructure layer (by collaborating with the orchestrator). We redesign the Scanflow ML framework to support such multiagent approach by using triggers, primitives, and strategies. We also implement a practical platform, so-called Scanflow-K8s, that enables autonomic ML workflows on Kubernetes clusters based on the Scanflow agents. MNIST image classification and MLPerf ImageNet classification benchmarks are used as case studies to show the capabilities of Scanflow-K8s under different scenarios. The experimental results demonstrate the feasibility and effectiveness of our proposed agent approach and the Scanflow-K8s platform for the autonomic management of ML workflows in Kubernetes clusters at multiple layers.

show abstract

Section: B Agent Social Abilitymentioning

confidence: 99%

Section: B Mnist Classificationmentioning

confidence: 99%

Section: B Mnist Classificationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Scanflow-K8s: Agent-based Framework for Autonomic Management and Supervision of ML Workflows in Kubernetes Clusters

Liu

Bravo-Rocca

Guitart

et al. 2022

2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The internal behavior of the agents in this paper builds on a previous work [29] that considers model debugging through static nodes in a graph. In a nutshell, we propose to convert those static nodes into dynamic agents (the Checker and Improver) and add two agents with new responsibilities (the Supervisor and Planner).…”

Section: Architecture Of the Multi-agent Systemmentioning

confidence: 99%

Human-in-the-loop online multi-agent approach to increase trustworthiness in ML models through trust scores and data augmentation

Bravo-Rocca¹,

Liu²,

Guitart³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Increasing a ML model accuracy is not enough, we must also increase its trustworthiness. This is an important step for building resilient AI systems for safety-critical applications such as automotive, finance, and healthcare. For that purpose, we propose a multi-agent system that combines both machine and human agents. In this system, a checker agent calculates a trust score of each instance (which penalizes overconfidence and overcautiousness in predictions) using an agreement-based method and ranks it; then an improver agent filters the anomalous instances based on a human rule-based procedure (which is considered safe), gets the human labels, applies geometric data augmentation, and retrains with the augmented data using transfer learning. We evaluate the system on corrupted versions of the MNIST and FashionMNIST datasets. We get an improvement in accuracy and trust score with just few additional labels compared to a baseline approach.

show abstract

An empirical study of challenges in machine learning asset management

Zhao,

Chen,

Bangash

et al. 2024

Empir Software Eng

View full text Add to dashboard Cite

Scanflow: A multi-graph framework for Machine Learning workflow management, supervision, and debugging

Cited by 9 publications

References 26 publications

Scanflow-K8s: Agent-based Framework for Autonomic Management and Supervision of ML Workflows in Kubernetes Clusters

Scanflow-K8s: Agent-based Framework for Autonomic Management and Supervision of ML Workflows in Kubernetes Clusters

Human-in-the-loop online multi-agent approach to increase trustworthiness in ML models through trust scores and data augmentation

An empirical study of challenges in machine learning asset management

Contact Info

Product

Resources

About