Workflow provenance in the lifecycle of scientific machine learning

Abstract

Machine learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibility, model explainability, and experiment data understanding. However, scientific ML is multidisciplinary, heterogeneous, and affected by the physical constraints of the domain, making such analyses even more challenging. In this work, we leverage workflow provenance techniques to build a holistic view to support the lifecycle of scientific ML. We contribute with (i) characterization of the lifecycle and taxonomy for data analyses; (ii) design decisions to build this view, with a W3C PROV compliant data representation and a reference system architecture; and (iii) lessons learned after an evaluation in an Oil & Gas case using an HPC cluster with 393 nodes and 946 GPUs. The experiments show that the decisions enable queries that integrate domain semantics with ML models while keeping low overhead (<1%), high scalability, and an order of magnitude of query acceleration under certain workloads against without our representation.

Building Trust in Earth Science Findings through Data Traceability and Results Explainability

“…Figure 12 shows the results of the bandwidth for the three IO sizes: (a) 1 GB, (b) 10 GB, and (c) 80 GB. The xaxis shows the number of files [1,10,100, 1000] and the y-axis shows the measured bandwidth in MB/s. For each number of files we measure the read and write IO 5 times for both environments.…”

Life Science Workflow Services (LifeSWS): Motivations and Architecture

“…LifeSWS capitalizes on our previous experience in developing major systems for scientific applications such as: polystores with CloudMdSQL [17], workflows with OpenAlea [30], model management with Gypscie [35] [38], querying data across distributed services with DfAnalyzer [32] and Provlake [33], monitoring and debugging applications implemented in big data frameworks such as Apache Spark [12], and debugging workflows with BugDoc [18] and VersionClimber [29].…”

“…It provides APIs to access ML development tools, such as PyTorch, Scikit-learn and Tensorflow. Also motivated by the objective of providing a holistic view to support the lifecycle of scientific ML, ProvLake [33] is a provenance data management system capable of capturing, integrating, and querying data across multiple distributed services, programs, databases, stores, and computational workflows by leveraging provenance data.…”

Augmented lineage: traceability of data analysis including complex UDF processing

“…Recently, some researches targeted the distributed big data processing systems and proposed how to capture provenance (e.g., MapReduce [2,36], Spark [30] and Flink [41,42]). In addition, provenance has been also researched in the ML domain, and [39,48,54] focused on the provenance for the training phase of ML models.…”