A comparative study of existing quality measures for process discovery

Abstract

Evaluating the quality of discovered process models is an important task in many process mining analyses. Currently, several metrics measuring the fitness, precision and generalization of a discovered model are implemented. However, there is little empirical evidence how these metrics relate to each other, both within and across these different quality dimensions. In order to better understand these relationships, a large-scale comparative experiment was conducted. The statistical analysis of the results shows that, although fitness and precision metrics behave very similar within their dimension, some are more pessimistic while others are more optimistic. Furthermore, it was found that there is no agreement between generalization metrics. The results of the study can be used to inform decisions on which quality metrics to use in practice. Moreover, they highlight issues which give rise to new directions for future research in the area of quality measurement.

Towards Confirmatory Process Discovery: Making Assertions About the Underlying System

“…Over the last decades, several metrics have been implemented to measure these quality dimensions. For a comprehensive overview of these metrics, we refer to Table 1 and Janssenswillen et al (2017). The state-of-theart metrics will be further introduced in Sect.…”

“…A recent comparative study of process metrics (Janssenswillen et al 2017) shows that the role of generalization in measuring conformance is extremely ambiguous. The generalization metrics were found to be uncorrelated, with one of them appearing to be related to fitness.…”

A process mining methodology for modeling unstructured processes

“…1) Accuracy is quantitatively estimated through fitness and precision (Buijs, van Dongen, & van der Aalst, 2012; De Weerdt et al, 2012; Janssenswillen, Donders, Jouck, & Depaire, 2017; van der Aalst, 2016; van der Aalst, Adriansyah, de Medeiros, et al, 2012). Based on extant studies, fitness measures the extent to which the discovered model accurately reproduces the cases recorded in the log.…”

“…2) Comprehensibility is typically evaluated through model simplicity (Buijs et al, 2012; Janssenswillen et al, 2017; Mendling, Reijers, & Cardoso, 2007; van der Aalst, 2016): the simplest model capable to describe the process is the best model. Mathematically, simplicity measures the number of elements in the Petri Net models.…”

Cross-domain similarity assessment for workflow improvement to handle Big Data challenge in workflow management

“…After improving the workflow the coupling of the improved workflow is computed (Table 4). Simplified models of workflows can be defined as models which are not overly complex, e.g., they are not extremely large and the density of arcs and coupling of elements is low [42,43]. For a process that consists of a set of task tags (S) on the workflow structure the process coupling (cp) [42] is defined as follows:…”