Monotone Precision and Recall Measures for Comparing Executions and Specifications of Dynamic Systems

Abstract

The behavioural comparison of systems is an important concern of software engineering research. For example, the areas of specification discovery and specification mining are concerned with measuring the consistency between a collection of execution traces and a program specification. This problem is also tackled in process mining with the help of measures that describe the quality of a process specification automatically discovered from execution logs. Though various measures have been proposed, it was recently demonstrated that they neither fulfil essential properties, such as monotonicity , nor can they handle infinite behaviour. In this article, we address this research problem by introducing a new framework for the definition of behavioural quotients. We prove that corresponding quotients guarantee desired properties that existing measures have failed to support. We demonstrate the application of the quotients for capturing precision and recall measures between a collection of recorded executions and a system specification. We use a prototypical implementation of these measures to contrast their monotonic assessment with measures that have been defined in prior research.

Monotone Conformance Checking for Partially Matching Designed and Observed Processes

“…Because topological entropy is defined over irreducible languages, one cannot use it to estimate cardinality of infinite non-irreducible regular languages or finite languages, e.g., event logs. To allow estimating cardinality of an arbitrary regular language, in [8], we proposed the notion of a shortcircuit measure over languages. Given a measure over languages m : [8], ent•(L) can always be computed.…”

“…To allow estimating cardinality of an arbitrary regular language, in [8], we proposed the notion of a shortcircuit measure over languages. Given a measure over languages m : [8], ent•(L) can always be computed.…”

“…Process discovery studies methods, techniques, and tools to automatically construct a "good" process model from an event log to aggregate and analyze the knowledge about business processes as they were observed in the real world [2]- [4]. Conformance checking studies methods, techniques, and tools to characterize and quantify differences between an event log and process model [5]- [8], e.g., to measure and explain the "goodness" of the model (constructed or discovered) that encodes the traces from the event log.…”

“…This makes it practically difficult to define conformance measures with the desired properties, e.g., determinism and monotonicity, as it is not immediate to define how to measure infinite collections of traces. In [8], we overcame this challenge by proposing process conformance measures of precision and recall based on the notion of topological entropy over regular languages.…”

“…• Extends the entropy-based precision and recall measures proposed in [8] to support the partial matching between traces via comparisons of their sub-traces; • Introduces, for the first time, properties for conformance measures that address the partial matching of traces; note that all the so far introduced properties for conformance measures [8], [10], [11] only address the comparisons of identical traces; and • Reports on qualitative and quantitative evaluations of the proposed precision and recall measures that demonstrate the feasibility of using them in industrial settings.…”