Job life cycle management libraries for CMS workflow management projects

Abstract

Scientific analysis and simulation requires the processing and generation of millions of data samples. These processing and generation tasks are often comprised of multiple smaller tasks divided over multiple (computing) sites. This paper discusses the Compact Muon Solenoid (CMS) workflow infrastructure, and specifically the Python based workflow library which is used for so called task lifecycle management. The CMS workflow infrastructure consists of three layers: high level specification of the various tasks based on input/output datasets, life cycle management of task instances derived from the high level specification and execution management. The workflow library is the result of a convergence of three CMS subprojects that respectively deal with scientific analysis, simulation and real time data aggregation from the experiment.

CMS Computing: Performance and Outlook

“…A new workflow management system, WMAgent [7], is much more robust against such problems, and its deployment was a great help to operations. It is a state machine rather than a messaging system, and has 100% accountability for all events processed.…”

User-centric monitoring of the analysis and production activities within the ATLAS and CMS Virtual Organisations using the Experiment Dashboard system

“…In CMS, there are at least five different submission tools and most of them provide their own monitoring system, for example, CRAB Monitoring is coupled to jobs submitted by the CRAB submission system [3] and WMAgent Monitoring is coupled to jobs submitted via the WMAgent workflow management tool [4]. Similarly, in ATLAS, there are two different workload submission systems that provide their own monitoring applications.…”