Vamsa: Automated Provenance Tracking in Data Science Scripts

Namaki, Mohammad Hossein; Floratou, Avrilia; Psallidas, Fotis; Krishnan, Subru; Agrawal, Ashvin; Wu, Yinghui; Zhu, Yiwen; Weimer, Markus

doi:10.1145/3394486.3403205

Cited by 33 publications

(11 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Metadata generated by the model during training or testing are extremely important for model debugging. Auxiliary debugging tools [21][22][23] or data provenance studies [30][31][32] extract metadata using a data collection API during training. However, in the federated learning setting, manual extraction and analysis of training metadata are not allowed because of the data protection requirement.…”

Section: Metadata Capturementioning

confidence: 99%

Fed-DNN-Debugger: Automatically Debugging Deep Neural Network Models in Federated Learning

Duan

Liu

Han

et al. 2023

Security and Communication Networks

View full text Add to dashboard Cite

Federated learning is a distributed machine learning framework that has been widely applied in scenarios that require data privacy. To obtain a neural network model that performs well, when the model falls into a bug, existing solutions retrain it on a larger training dataset or the carefully selected samples from model diagnosis. To overcome this challenge, this paper presents Fed-DNN-Debugger, which can automatically and efficiently fix DNN models in federated learning. Fed-DNN-Debugger fixes the federated model by fixing each client model. Fed-DNN-Debugger consists of two modules for debugging a client model: nonintrusive metadata capture (NIMC) and automated neural network model debugging (ANNMD). NIMC collects the metadata with deep learning software syntax automatically. It does not insert any code for metadata collection into modeling scripts. ANNMD scores samples according to metadata and searches for high-quality samples. Models are retrained on the selected samples to repair their weights. Our experiments with popular federated models show that Fed-DNN-Debugger can improve the test accuracy by 8% by automatically fixing models.

show abstract

Section: Metadata Capturementioning

confidence: 99%

Fed-DNN-Debugger: Automatically Debugging Deep Neural Network Models in Federated Learning

Duan

Liu

Han

et al. 2023

Security and Communication Networks

View full text Add to dashboard Cite

show abstract

“…By contrast, our approach automatically refreshes dependent cells. Vamsa [6] also employes static dataflow analysis to analyze provenance of Python ML pipelines. Dataflow notebooks [4] extend Jupyter with immutable identifiers for cells and the capability to reference the results of a cell by its identifier.…”

Section: Related Workmentioning

confidence: 99%

Runtime provenance refinement for notebooks

Deo¹,

Glavic

Kennedy

2022

Proceedings of the 14th International Workshop on the Theory and Practice of Provenance

View full text Add to dashboard Cite

Computational notebooks (e.g., Jupyter or Apache Zeppelin) have become a popular choice for data exploration, preparation, and ETL. Notebooks are more suited for interactive development of data pipelines than classical workflow systems, because they provide immediate feedback for the results of a computation and do not require the full computation to be specified upfront. However, the notebook model suffers from poor reproducibility, does not support automatic incremental re-evaluation of code when the code or inputs change, and does not allow for parallel execution of cellsall symptoms of its kernel-based evaluation strategy. We propose a new "workbook" model that combines the usability of notebooks with the provenance and parallel execution capabilities of workflow systems. This is made possible through a novel approach that refines a static approximation of provenance for Python code at runtime and a scheduler that dynamically adapts the execution order of cells based on data dependencies detected or refuted at runtime. We demonstrate the feasibility of this approach using a prototype implementation in our notebook engine Vizier.

show abstract

“…The interest in workflow provenance management has increased in the recent years, driven by a major effort by the provenance community, 31,35,[50][51][52][53][54][55][56][57][58][59][60][61][62] particularly to explore possibilities of optimizing workflows with the data captured by provenance tools and as a response to the urgent need for reproducible science, which is critical in scientific ML. 63 To exemplify, Thavasimani et al 14 investigate provenance traces recorded during workflow executions to observe differences in results with minor workflow configuration differences.…”

Section: Related Workmentioning

confidence: 99%

Workflow provenance in the lifecycle of scientific machine learning

Souza¹,

Azevedo²,

Lourenço³

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

Machine learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibility, model explainability, and experiment data understanding. However, scientific ML is multidisciplinary, heterogeneous, and affected by the physical constraints of the domain, making such analyses even more challenging. In this work, we leverage workflow provenance techniques to build a holistic view to support the lifecycle of scientific ML. We contribute with (i) characterization of the lifecycle and taxonomy for data analyses; (ii) design decisions to build this view, with a W3C PROV compliant data representation and a reference system architecture; and (iii) lessons learned after an evaluation in an Oil & Gas case using an HPC cluster with 393 nodes and 946 GPUs. The experiments show that the decisions enable queries that integrate domain semantics with ML models while keeping low overhead (<1%), high scalability, and an order of magnitude of query acceleration under certain workloads against without our representation.

show abstract

Vamsa: Automated Provenance Tracking in Data Science Scripts

Cited by 33 publications

References 16 publications

Fed-DNN-Debugger: Automatically Debugging Deep Neural Network Models in Federated Learning

Fed-DNN-Debugger: Automatically Debugging Deep Neural Network Models in Federated Learning

Runtime provenance refinement for notebooks

Workflow provenance in the lifecycle of scientific machine learning

Contact Info

Product

Resources

About