Monitoring a CI/CD Workflow Using Process Mining

Abstract

Process mining (PM) is a unique approach to extract workflow models of actual real-world activities, namely those related to software development. To be efficient and produce more reliable results, its algorithms require structured input data. However, actual real-world data originate from multiple heterogeneous sources; thus, integration and normalization are required preparatory steps before applying PM techniques. This problem is exacerbated by the need of performing this analysis in real time, rather than off-line in a batch-style approach. In this paper, we show how Apache Kafka pipelines can be used to support the integration and normalization of the event logs from multiple sources into data streams that feed the process mining algorithms in real-time. An application to the complex CI/CD pipeline of a major European e-commerce company is presented, showing that these techniques provide means to monitor and have higher observability of development processes.

Special Issue on New Paradigms of Software Production and Deployment

“…Authors propose a model-driven workflow to cope with these challenges, which is illustrated and validated via an open-source case study. Monitoring a CI/CD Workflow Using Process Mining , by [ 8 ]: this papers describes how data generated by the execution of CI/CD pipelines can be used to monitor (and then gain better understanding) about how the current pipelines are working. The approach proposed by the authors relies on the implementation of process mining algorithms into a well known open source distributed streaming platform.…”

PM4Py-GPU: a High-Performance General-Purpose Library for Process Mining

“…The drawback of Spark is the additional overhead due to the log distribution step, which limits the performance benefits of the platform. Other platform such as Apache Kafka have been used for processing of streams [5]. Application-tailored engines have also been proposed.…”