Measuring precision of modeled behavior

Abstract

Conformance checking techniques compare observed behavior (i.e., event logs) with modeled behavior for a variety of reasons. For example, discrepancies between a normative process model and recorded behavior may point to fraud or inefficiencies. The resulting diagnostics can be used for auditing and compliance management. Conformance checking can also be used to judge a process model automatically discovered from an event log. Models discovered using different process discovery techniques need to be compared objectively. These examples illustrate just a few of the many use cases for aligning observed and modeled behavior. Thus far, most conformance checking techniques focused on replay fitness, i.e., the ability to reproduce the event log. However, it is easy to construct models that allow for lots of behavior (including the observed behavior) without being precise.In this paper, we propose a method to measure precision of process models, given their event logs by first aligning the logs to the models. This way, the measurement is not sensitive to non-fitting executions and more accurate values can be obtained for non-fitting logs. Furthermore, we introduce several variants of the technique to deal better with incomplete logs and reduce possible bias due to behavioral property of process models. The approach has been implemented in the ProM 6 framework and tested against both artificial and real-life cases. Experiments show that the approach is robust to noise and applicable to handle logs and models of real-life complexity.

Scientific workflows for process mining: building blocks, scenarios, and implementation

“…In this experiment, we implemented a process mining workflow using RapidProM to extract the model that scores higher with respect to the geometric average of precision and replay fitness. 2 The geometric average of replay fitness and precision seems to be better than the arithmetic average since it is necessary to have a strong penalty if one of the criteria is low. For this experiment, we employed the Inductive MinerInfrequent discovery technique [29] and used different values for the noise threshold parameter.…”

“…Process mining aims to bridge the gap between BPM and WfM on the one hand and DM, BI, and ML on the other hand. A wealth of process discovery [29,53,62] and conformance checking [1,2,48] techniques has become available. For example, the process mining framework ProM [58] provides hundreds of plug-ins supporting different types of process mining (http://www.processmining.org).…”

Anti-alignments in Conformance Checking – The Dark Side of Process Models

“…The modified high-level overview is shown in Figure 2. Current metrics for precision (e.g., [5]) will not consider this modification as a severe one: the precision of the model with respect to the log will be very similar before or after the modification.…”

“…In [4,5] the metric align precision (a p ) was presented to estimate the precision a process model N (a Petri net) has in characterizing observed behavior, described by an event log L. Informally the computation of a p is as follows: for each trace σ from the event log, a trace γ in the model which has minimal number of deviations with respect to σ is computed (denoted by γ ∈ Γ (σ, N )), by using the techniques from [3] 3 . Let …”

“…For instance, a highly-deviating anti-alignment may be a sign of a loss in precision, which can be missed by current metrics as they bound considerably the exploration of model state space for the sake of efficiency [5].…”