Building Continuous Integration Services for Machine Learning

Karlaš, Bojan; Interlandi, Matteo; Renggli, Cédric; Wu, Wentao; Zhang, Ce; Babu, Deepak Mukunthu Iyappan; Edwards, Jordan; Lauren, Chris; Xu, Y.; Weimer, Markus

doi:10.1145/3394486.3403290

Cited by 19 publications

(9 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The key ingredients of our system lie in (a) the syntax and semantics of the test conditions and how to accurately evaluate them, and (b) an optimized sample-size estimator that yields a budget of test set re-uses before it needs to be refreshed. For a full description of the workflow as well as advanced system optimizations deployed in our engine, we refer the reader to our initial paper [31] and the followup work [22], which further discusses the integration into existing software development ecosystems.…”

Section: Mlops Challengementioning

confidence: 99%

A Data Quality-Driven View of MLOps

Renggli,

Rimanic,

Gürel

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Developing machine learning models can be seen as a process similar to the one established for traditional software development. A key difference between the two lies in the strong dependency between the quality of a machine learning model and the quality of the data used to train or perform evaluations. In this work, we demonstrate how different aspects of data quality propagate through various stages of machine learning development. By performing a joint analysis of the impact of well-known data quality dimensions and the downstream machine learning process, we show that different components of a typical MLOps pipeline can be efficiently designed, providing both a technical and theoretical perspective.

show abstract

Section: Mlops Challengementioning

confidence: 99%

A Data Quality-Driven View of MLOps

Renggli,

Rimanic,

Gürel

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Prospective collaborators may submit code to a shared codebase. Some code may introduce errors or decrease the performance of the ML model [46,48,69,81]. How can code contributions be evaluated?…”

Section: Challengesmentioning

confidence: 99%

Enabling Collaborative Data Science Development with the Ballet Framework

Smith

Cito

et al. 2021

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

While the open-source software development model has led to successful large-scale collaborations in building software systems, data science projects are frequently developed by individuals or small teams. We describe challenges to scaling data science collaborations and present a conceptual framework and ML programming model to address them. We instantiate these ideas in Ballet, the first lightweight framework for collaborative, open-source data science through a focus on feature engineering, and an accompanying cloud-based development environment. Using our framework, collaborators incrementally propose feature definitions to a repository which are each subjected to software and ML performance validation and can be automatically merged into an executable feature engineering pipeline. We leverage Ballet to conduct a case study analysis of an income prediction problem with 27 collaborators, and discuss implications for future designers of collaborative projects.CCS Concepts: • Human-centered computing → Collaborative and social computing systems and tools; Empirical studies in collaborative and social computing; • Computing methodologies → Machine learning; • Software and its engineering → Collaboration in software development.

show abstract

“…Although all the project under this study are written using Python, no project is using any common tool that integrates the DS modules and provides interface to the pipeline. Today, continuous integration and deployment (CI/CD) tools are widely used as a common practice in traditional software lifecycle to automate compilation, building, and testing [26,37]. Additionally, from our subject studies of pipelines in theory, we found some CI/CD tools designed for ML pipelines are available [27,47,49].…”

Section: Characteristics Of Pipelines In-the-largementioning

confidence: 99%

“…Although CI/CD frameworks e.g., TravisCI, GitHub Actions, Microsoft Azure DevOps are well established for traditional software such as web applications, several challenges remain for DS pipelines. Karlaš et al outlined a major CI/CD challenge for DS pipeline i.e., unlike traditional software, ML testing is probabilistic [37]. The authors also pointed out to the gap between recent theoretical development of CI/CD in DS and their usage in practice.…”

Section: Characteristics Of Pipelines In-the-largementioning

confidence: 99%

The Art and Practice of Data Science Pipelines: A Comprehensive Study of Data Science Pipelines In Theory, In-The-Small, and In-The-Large

Biswas,

Wardat,

Rajan

2021

Preprint

View full text Add to dashboard Cite

Increasingly larger number of software systems today are including data science components for descriptive, predictive, and prescriptive analytics. The collection of data science stages from acquisition, to cleaning/curation, to modeling, and so on are referred to as data science pipelines. To facilitate research and practice on data science pipelines, it is essential to understand their nature. What are the typical stages of a data science pipeline? How are they connected? Do the pipelines differ in the theoretical representations and that in the practice? Today we do not fully understand these architectural characteristics of data science pipelines. In this work, we present a three-pronged comprehensive study to answer this for the stateof-the-art, data science in-the-small, and data science in-the-large. Our study analyzes three datasets: a collection of 71 proposals for data science pipelines and related concepts in theory, a collection of over 105 implementations of curated data science pipelines from Kaggle competitions to understand data science in-the-small, and a collection of 21 mature data science projects from GitHub to understand data science in-the-large. Our study has led to three representations of data science pipelines that capture the essence of our subjects in theory, in-the-small, and in-the-large. CCS CONCEPTS• Software and its engineering → Software creation and management; • Computing methodologies → Machine learning.

show abstract

Building Continuous Integration Services for Machine Learning

Cited by 19 publications

References 12 publications

A Data Quality-Driven View of MLOps

A Data Quality-Driven View of MLOps

Enabling Collaborative Data Science Development with the Ballet Framework

The Art and Practice of Data Science Pipelines: A Comprehensive Study of Data Science Pipelines In Theory, In-The-Small, and In-The-Large

Contact Info

Product

Resources

About