On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects

Barrak, Amine; Eghan, Ellis E.; Adams, Bram

doi:10.1109/saner50967.2021.00046

Cited by 22 publications

(10 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are some tools available for supporting monitoring data quality tools such as Great Expectations [37], Databand [38], and Dataform [39]. Additionally, MLFlow [40] and Data Version Control (DVC) [41] also support data collection and management for model experiments, datasets, and parameters. However, these frameworks do not support a holistic explainability approach by keeping track of multiaspect requirements discussed in Section III concerning endto-end ML.…”

Section: Discussion On Tooling and Integrationmentioning

confidence: 99%

Holistic Explainability Requirements for End-to-End Machine Learning in IoT Cloud Systems

Nguyen

Phung²,

Ly³

et al. 2021

2021 IEEE 29th International Requirements Engineering Conference Workshops (REW)

View full text Add to dashboard Cite

End-to-end machine learning (ML) in Internet of Things (IoT) Cloud systems consists of multiple processes, covering data, model, and service engineering, and involves multiple stakeholders. Therefore, to be able to explain ML to relevant stakeholders, it is important to identify explainability requirements in a holistic manner. In this paper, we present our methodology to address explainability requirements for endto-end ML in developing ML services to be deployed within IoT Cloud systems. We identify and classify explainability requirements engineering through (i) involvement of relevant stakeholders, (ii) end-to-end data, model, and service engineering processes, and (iii) multiple explainability aspects. We present our work with a case study of predictive maintenance for Base Transceiver Stations (BTS) in the telco domain.

show abstract

Section: Discussion On Tooling and Integrationmentioning

confidence: 99%

Holistic Explainability Requirements for End-to-End Machine Learning in IoT Cloud Systems

Nguyen

Phung²,

Ly³

et al. 2021

2021 IEEE 29th International Requirements Engineering Conference Workshops (REW)

View full text Add to dashboard Cite

show abstract

“…While considering the stated demands on data quality during the various sections of the work on hand, no specific framework or quality management model for big data value chains, as introduced by the authors, is proposed. Barrak et al empirically analyzed 391 open-source projects which used Data Version Control (DVC) techniques with respect to coupling of software and DVC artifacts and their complexity evolution in [18]. Their empirical study concludes that using DVC versioning tools becomes a growing practice, even though there is a maintenance overhead.…”

Section: Related Work and State-of-the-artmentioning

confidence: 99%

“…As there is often a massive amount of frequently changing input data involved in ML projects, local data repositories (e.g., edge devices) often hold the actual data set. At the same time, the remote storage solutions only persist a hash of the current version [18].…”

Section: Data Versioningmentioning

confidence: 99%

Demystifying MLOps and Presenting a Recipe for the Selection of Open-Source Tools

et al. 2021

View full text Add to dashboard Cite

Nowadays, machine learning projects have become more and more relevant to various real-world use cases. The success of complex Neural Network models depends upon many factors, as the requirement for structured and machine learning-centric project development management arises. Due to the multitude of tools available for different operational phases, responsibilities and requirements become more and more unclear. In this work, Machine Learning Operations (MLOps) technologies and tools for every part of the overall project pipeline, as well as involved roles, are examined and clearly defined. With the focus on the inter-connectivity of specific tools and comparison by well-selected requirements of MLOps, model performance, input data, and system quality metrics are briefly discussed. By identifying aspects of machine learning, which can be reused from project to project, open-source tools which help in specific parts of the pipeline, and possible combinations, an overview of support in MLOps is given. Deep learning has revolutionized the field of Image processing, and building an automated machine learning workflow for object detection is of great interest for many organizations. For this, a simple MLOps workflow for object detection with images is portrayed.

show abstract

“…There are generally four types of assets to manage in machine learning (ML) in order to achieve model reproducibility: resources (e.g., dataset and environment), software (e.g., source code), metadata (e.g., dependencies), and execution data (e.g., execution results) [43]. However, prior work [23,25,43] shows these assets should not be managed with the same toolsets (e.g., Git) used for source code [23]. Hence, new version management tools (e.g., DVC [12] and MLflow [1]) are specifically designed for managing ML assets.…”

Section: Introductionmentioning

confidence: 99%

Towards Training Reproducible Deep Learning Models

Chen,

Wen,

Shi

et al. 2022

Preprint

View full text Add to dashboard Cite

Reproducibility is an increasing concern in Artificial Intelligence (AI), particularly in the area of Deep Learning (DL). Being able to reproduce DL models is crucial for AI-based systems, as it is closely tied to various tasks like training, testing, debugging, and auditing. However, DL models are challenging to be reproduced due to issues like randomness in the software (e.g., DL algorithms) and non-determinism in the hardware (e.g., GPU). There are various practices to mitigate some of the aforementioned issues. However, many of them are either too intrusive or can only work for a specific usage context. In this paper, we propose a systematic approach to training reproducible DL models. Our approach includes three main parts: (1) a set of general criteria to thoroughly evaluate the reproducibility of DL models for two different domains, (2) a unified framework which leverages a record-and-replay technique to mitigate software-related randomness and a profile-and-patch technique to control hardware-related non-determinism, and (3) a reproducibility guideline which explains the rationales and the mitigation strategies on conducting a reproducible training process for DL models. Case study results show our approach can successfully reproduce six open source and one commercial DL models. CCS CONCEPTS• Software and its engineering → Empirical software validation.

show abstract

On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects

Cited by 22 publications

References 23 publications

Holistic Explainability Requirements for End-to-End Machine Learning in IoT Cloud Systems

Holistic Explainability Requirements for End-to-End Machine Learning in IoT Cloud Systems

Demystifying MLOps and Presenting a Recipe for the Selection of Open-Source Tools

Towards Training Reproducible Deep Learning Models

Contact Info

Product

Resources

About