Managing Machine Learning Workflow Components

Moreno, Márcio Ferreira; Lourenço, Vítor; Fiorini, Sandro Rama; Costa, Polyana; Brandão, Rafael; Civitarese, Daniel; Cerqueira, Renato

doi:10.1109/icsc.2020.00011

Cited by 4 publications

(2 citation statements)

References 12 publications

(7 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Samuel 71 propose ProvBook, for reproducibility of ML experiments using Jupyter notebooks applying FAIR data principles. Moreno et al proposed MLWfM 72 to provide data concepts for ML and domain‐specific awareness, but without provenance concepts and a data representation. Brandao et al 73 proposed a knowledge‐based workflow management approach aiming at broadening user collaboration over ML experiments.…”

Section: Related Workmentioning

confidence: 99%

Workflow provenance in the lifecycle of scientific machine learning

Souza¹,

Azevedo²,

Lourenço³

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Machine learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibility, model explainability, and experiment data understanding. However, scientific ML is multidisciplinary, heterogeneous, and affected by the physical constraints of the domain, making such analyses even more challenging. In this work, we leverage workflow provenance techniques to build a holistic view to support the lifecycle of scientific ML. We contribute with (i) characterization of the lifecycle and taxonomy for data analyses; (ii) design decisions to build this view, with a W3C PROV compliant data representation and a reference system architecture; and (iii) lessons learned after an evaluation in an Oil & Gas case using an HPC cluster with 393 nodes and 946 GPUs. The experiments show that the decisions enable queries that integrate domain semantics with ML models while keeping low overhead (<1%), high scalability, and an order of magnitude of query acceleration under certain workloads against without our representation.

show abstract

Section: Related Workmentioning

confidence: 99%

Workflow provenance in the lifecycle of scientific machine learning

Souza¹,

Azevedo²,

Lourenço³

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…Thus, we propose another aspect for building a context from the bottom up and separating it into a similar method of abstraction, while focusing on generating it as a knowledge base. For solving the issues or hurdles mentioned above, studies with Automated Machine Learning (AutoML) 34 and workflow components for machine learning (ML) applications 35,36 have been conducted. Particularly Auto-WEKA 37 or auto-sklearn 38 are the ones of AutoML concepts to automate analysis tasks.…”

Section: Related Workmentioning

confidence: 99%

A method to generate context information sets from analysis results with a unified abstraction model based on an extension of data enrichment scheme

Park

Mun

Choi

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

Context-awareness techniques generally deal with steps for processing information that can be gathered from perspectives of objects in real-world scenes in accordance with resolving heterogeneous devices and providing meaningful context information in recent computing domains such as Internet of Things, cloud computing, edge computing, and big data analysis. However, analysis applications are necessary to be improved in a way of cooperative approaches because they are independently executed far away from context-aware systems. This means that the stages are applied to process the data or the derived results, are isolated. That leads difficulties of management and integration for other software modules. As a result, a computing paradigm called edge computing has been suggested to compromise such isolations. For processing context information, it is necessary to establish knowledge parts with validation processes against the isolations. In this paper, we suggest a method for processing datasets in a view of building knowledge parts. The suggested method uses an extensible data enrichment scheme, which represents the relations for the object. In the experiment section, applications of the proposed method are shown in one of the healthcare community centers, and the data abstraction with the analysis jobs can be processed for each real-world object. K E Y W O R D Scontext-awareness, data abstraction, edge computing environment, embedding convergence, knowledge representation INTRODUCTIONRecent improvements in computational methodologies and technologies such as cloud computing, Internet of Things (IoT), 5G networks, and big data have enabled to build meaningful knowledge for artificial intelligence. [1][2][3][4][5] Among these improvements, there has been innovative approaches to develop the subsets of an architecture for application in computation domains. It is important to consider that bringing the advantages provides benefits to lives for human being and society for transformative innovation. [6][7][8] Because recent trends and challenges have focused on processing massive amounts of data, also known as big data, so that different approaches related to the collection, management, and application of such datasets have been conducted. Therefore, it is necessary to design the stages that are highly associated with handling these datasets. 9,10 As the result, both processing and providing meaningful context information are important techniques where massive data can be generated on state-of-the-art domains such as IoT, edge computing, or big data analysis. For context-aware systems, despite combining the key techniques mentioned above, there remain certain difficulties to overcome. [11][12][13] In general, context-awareness consists of resolving hardware heterogeneity, data acquisition, collection of

show abstract