Scalable Declarative HEP Analysis Workflows for Containerised Compute Clouds

Abstract

We describe a novel approach for experimental High-Energy Physics (HEP) data analyses that is centred around the declarative rather than imperative paradigm when describing analysis computational tasks. The analysis process can be structured in the form of a Directed Acyclic Graph (DAG), where each graph vertex represents a unit of computation with its inputs and outputs, and the graph edges describe the interconnection of various computational steps. We have developed REANA, a platform for reproducible data analyses, that supports several such DAG workflow specifications. The REANA platform parses the analysis workflow and dispatches its computational steps to various supported computing backends (Kubernetes, HTCondor, Slurm). The focus on declarative rather than imperative programming enables researchers to concentrate on the problem domain at hand without having to think about implementation details such as scalable job orchestration. The declarative programming approach is further exemplified by a multi-level job cascading paradigm that was implemented in the Yadage workflow specification language. We present two recent LHC particle physics analyses, ATLAS searches for dark matter and CMS jet energy correction pipelines, where the declarative approach was successfully applied. We argue that the declarative approach to data analyses, combined with recent advancements in container technology, facilitates the portability of computational data analyses to various compute backends, enhancing the reproducibility and the knowledge preservation behind particle physics data analyses.

Data Preservation in High Energy Physics (DPHEP Global Report 2022)

“…The REANA approach was successfully tested in several data production and data analysis scenarios. For example, REANA was used for ATLAS reinterpretation searches for new physics or CMS reconstruction and jet energy corrections [93]. In the CERN Open Data portal, REANA is used both on the "data production" side to ensure the correctness of preserved data sets' provenance information [90] as well as on the "data analysis" side by running several data analysis examples such as CMS open data Higgs-to-four-lepton example analysis (see Fig.…”

“…The integration with source code management platforms such as GitLab allows researchers to develop analyses on GitLab and run either full data analysis tasks on REANA or at least test the correctness of analysis workflow after each code change. If an analysis is developed in this "continuous integration" manner [93], the preservation of knowledge associated with the data analysis as well as the future deposit of analysis assets into digital repositories are largely facil-itated. REANA therefore complements the data preservation repositories by promoting active "preproducibility" of data analyses during the active analysis phase rather than only relying on passive data deposition and subsequent "reproducibility" once the analysis is completed [94].…”

Jets and Jet Substructure at Future Colliders

“…This approach, however, integrates over a huge phase space and often yields suboptimal values for JSS [251]. With the help of declarative and therefore consistently repeatable workflows [254] and machine-learning techniques, this approach can be significantly improved. The generator settings can be adjusted iteratively by repeating dedicated JSS measurements, consequently yielding optimal settings for the given suite of measurements.…”

An intelligent Data Delivery Service for and beyond the ATLAS experiment

“…The support of loop workflows in iDDS makes it possible. Currently it's already integrated with PanDA and REANA [12]. It has successfully been tested with a mono-Hbb analysis.…”