Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology

Studer, Stefan; Bui, Thanh; Drescher, Christian; Hanuschkin, Alexander; Winkler, Ludwig; Peters, Steven; Müller, Klaus‐Robert

doi:10.20944/preprints202103.0135.v1

Cited by 25 publications

(39 citation statements)

References 71 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…53 In general, larger consistent data sets result in better ML models and reduce over-fitting. 54 Due to the fixed recording angle and flow orientation of the current data set, conventional data augmentation techniques cannot be applied to increase training variability. Specific flow field-related augmentations are difficult to define and might introduce artifacts in the artificially augmented samples.…”

Section: Discussionmentioning

confidence: 99%

Deep feature learning of in-cylinder flow fields to analyze cycle-to-cycle variations in an SI engine

Dreher

Schmidt

Welch

et al. 2020

International Journal of Engine Research

Self Cite

View full text Add to dashboard Cite

Machine learning (ML) models based on a large data set of in-cylinder flow fields of an IC engine obtained by high-speed particle image velocimetry allow the identification of relevant flow structures underlying cycle-to-cycle variations of engine performance. To this end, deep feature learning is employed to train ML models that predict cycles of high and low in-cylinder maximum pressure. Deep convolutional autoencoders are self-supervised-trained to encode flow field features in low dimensional latent space. Without the limitations ascribable to manual feature engineering, ML models based on these learned features are able to classify high energy cycles already from the flow field during late intake and the compression stroke as early as 290 crank angle degrees before top dead center ([Formula: see text]) with a mean accuracy above chance level. The prediction accuracy from [Formula: see text] to [Formula: see text] is comparable to baseline ML approaches utilizing an extensive set of engineered features. Relevant flow structures in the compression stroke are revealed by feature analysis of ML models and are interpreted using conditional averaged flow quantities. This analysis unveils the importance of the horizontal velocity component of in-cylinder flows in predicting engine performance. Combining deep learning and conventional flow analysis techniques promises to be a powerful tool for ultimately revealing high-level flow features relevant to the prediction of cycle-to-cycle variations and further engine optimization.

show abstract

Section: Discussionmentioning

confidence: 99%

Deep feature learning of in-cylinder flow fields to analyze cycle-to-cycle variations in an SI engine

Dreher

Schmidt

Welch

et al. 2020

International Journal of Engine Research

Self Cite

View full text Add to dashboard Cite

show abstract

“…The recently published CRISP-ML(Q) (Studer et al, 2020 ) proposes an incremental extension of CRISP-DM with the monitoring and maintenance phases. While the study mentions “model explainability” referring to the technical aspects of the underlying model, it does not consider interpretability and explainability in a systematic way as CRISP-ML (Kolyshkina and Simoff, 2019 ).…”

Section: Crisp-ml Methodology—toward Interpretability-centric Creation Of ML Solutionsmentioning

confidence: 99%

“…The methodology then ensures that participants establish the activities for each stakeholder group at each process stage that are required to achieve this level. CRISP-ML (Kolyshkina and Simoff, 2019 ) includes stages 3 and 4 (data predictive potential assessment and data enrichment in Figure 1 ), which are not present in CRISP-ML(Q) (Studer et al, 2020 ). As indicated in Kolyshkina and Simoff ( 2019 ), skipping these important phases can result in potential scope creep and even business project failure.…”

Section: Crisp-ml Methodology—toward Interpretability-centric Creation Of ML Solutionsmentioning

confidence: 99%

Interpretability of Machine Learning Solutions in Public Healthcare: The CRISP-ML Approach

Kolyshkina¹,

Simoff

2021

Front. Big Data

View full text Add to dashboard Cite

Public healthcare has a history of cautious adoption for artificial intelligence (AI) systems. The rapid growth of data collection and linking capabilities combined with the increasing diversity of the data-driven AI techniques, including machine learning (ML), has brought both ubiquitous opportunities for data analytics projects and increased demands for the regulation and accountability of the outcomes of these projects. As a result, the area of interpretability and explainability of ML is gaining significant research momentum. While there has been some progress in the development of ML methods, the methodological side has shown limited progress. This limits the practicality of using ML in the health domain: the issues with explaining the outcomes of ML algorithms to medical practitioners and policy makers in public health has been a recognized obstacle to the broader adoption of data science approaches in this domain. This study builds on the earlier work which introduced CRISP-ML, a methodology that determines the interpretability level required by stakeholders for a successful real-world solution and then helps in achieving it. CRISP-ML was built on the strengths of CRISP-DM, addressing the gaps in handling interpretability. Its application in the Public Healthcare sector follows its successful deployment in a number of recent real-world projects across several industries and fields, including credit risk, insurance, utilities, and sport. This study elaborates on the CRISP-ML methodology on the determination, measurement, and achievement of the necessary level of interpretability of ML solutions in the Public Healthcare sector. It demonstrates how CRISP-ML addressed the problems with data diversity, the unstructured nature of data, and relatively low linkage between diverse data sets in the healthcare domain. The characteristics of the case study, used in the study, are typical for healthcare data, and CRISP-ML managed to deliver on these issues, ensuring the required level of interpretability of the ML solutions discussed in the project. The approach used ensured that interpretability requirements were met, taking into account public healthcare specifics, regulatory requirements, project stakeholders, project objectives, and data characteristics. The study concludes with the three main directions for the development of the presented cross-industry standard process.

show abstract

“…BugDoc [21] looks at changes in a pre-processing pipeline that cause the models to fail, where high-level script and ordering is used to identify bad configurations. Others provide quality assurance frameworks [44] or embedded simulators to estimate fairness impacts of a particular pipeline [9]. Again, however, these are not geared for deep data introspection.…”

Section: Related Workmentioning

confidence: 99%

Capturing and querying fine-grained provenance of preprocessing pipelines in data science

et al. 2020

View full text Add to dashboard Cite

Data processing pipelines that are designed to clean, transform and alter data in preparation for learning predictive models, have an impact on those models' accuracy and performance, as well on other properties, such as model fairness. It is therefore important to provide developers with the means to gain an in-depth understanding of how the pipeline steps affect the data, from the raw input to training sets ready to be used for learning. While other efforts track creation and changes of pipelines of relational operators, in this work we analyze the typical operations of data preparation within a machine learning process, and provide infrastructure for generating very granular provenance records from it, at the level of individual elements within a dataset. Our contributions include: (i) the formal definition of a core set of preprocessing operators, and the definition of provenance patterns for each of them, and (ii) a prototype implementation of an application-level provenance capture library that works alongside Python. We report on provenance processing and storage overhead and scalability experiments, carried out over both real ML benchmark pipelines and over TCP-DI, and show how the resulting provenance can be used to answer a suite of provenance benchmark queries that underpin some of the developers' debugging questions, as expressed on the Data Science Stack Exchange.

show abstract

Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology

Cited by 25 publications

References 71 publications

Deep feature learning of in-cylinder flow fields to analyze cycle-to-cycle variations in an SI engine

Deep feature learning of in-cylinder flow fields to analyze cycle-to-cycle variations in an SI engine

Interpretability of Machine Learning Solutions in Public Healthcare: The CRISP-ML Approach

Capturing and querying fine-grained provenance of preprocessing pipelines in data science

Contact Info

Product

Resources

About