Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology

Abstract

Machine learning is an established and frequently used technique in industry and academia, but a standard process model to improve success and efficiency of machine learning applications is still missing. Project organizations and machine learning practitioners face manifold challenges and risks when developing machine learning applications and have a need for guidance to meet business expectations. This paper therefore proposes a process model for the development of machine learning applications, covering six phases from defining the scope to maintaining the deployed machine learning application. Business and data understanding are executed simultaneously in the first phase, as both have considerable impact on the feasibility of the project. The next phases are comprised of data preparation, modeling, evaluation, and deployment. Special focus is applied to the last phase, as a model running in changing real-time environments requires close monitoring and maintenance to reduce the risk of performance degradation over time. With each task of the process, this work proposes quality assurance methodology that is suitable to address challenges in machine learning development that are identified in the form of risks. The methodology is drawn from practical experience and scientific literature, and has proven to be general and stable. The process model expands on CRISP-DM, a data mining process model that enjoys strong industry support, but fails to address machine learning specific tasks. The presented work proposes an industry- and application-neutral process model tailored for machine learning applications with a focus on technical tasks for quality assurance.

Noise Immunity and Robustness Study of Image Recognition Using a Convolutional Neural Network

“…The low point density image recognition tasks have already been researched by the authors in a series of papers dedicated to the estimation and prediction of behavior of mobile communication network subscriber groups and complex clusters by analyzing teletraffic and geolocation data [ 23 , 24 , 25 ]. The locations of subscribers in groups can be considered as sparsely located points on images.…”

Sources of Risk of AI Systems

“…An assurance case is defined as a reasoned, verifiable artefact that supports the assertion that a set of overlying assertions are satisfied, including a systematic argument with underlying evidence and explicit assumptions to support those assertions [ 152 ]. However, these works lack a detailed list of concrete criteria and only describe a few cases at a time, such as fairness [ 151 ], or only structure them on the basis of life-cycle phases according to standards such as the CRISP-DM [ 153 ], which means that comprehensive coverage of relevant risk areas cannot be achieved [ 21 ]. International standards for the AI field are still under development and usually only address partial aspects, such as the explainability [ 23 ] or controllability [ 24 ], of these systems, which are not applicable to the field of safety-related systems [ 25 ].…”

Threats Detection during Human-Computer Interaction in Driver Monitoring Systems

“…Most of the modern approaches to analyzing driver state and identifying abnormal behavior are based on machine learning [ 8 , 9 , 10 ]. There is a large body of knowledge on different machine learning approaches; however, the basic schema of applying machine learning to solve a real-world problem is refined in the MLOps field (the AI domain has some fundamentally different aspects from both software development [ 11 ] and data mining [ 12 ] and, therefore, requires its own specific process). For example, the paper [ 12 ] proposes a process model for the development of machine learning applications, which covers six phases, from defining the scope, to maintaining the deployed machine learning application.…”

“…There is a large body of knowledge on different machine learning approaches; however, the basic schema of applying machine learning to solve a real-world problem is refined in the MLOps field (the AI domain has some fundamentally different aspects from both software development [ 11 ] and data mining [ 12 ] and, therefore, requires its own specific process). For example, the paper [ 12 ] proposes a process model for the development of machine learning applications, which covers six phases, from defining the scope, to maintaining the deployed machine learning application. The process model expands on CRISP-DM, a data mining process model that enjoys strong industry support but lacks the ability to address machine learning specific tasks.…”