Open Data and Data Analysis Preservation Services for LHC Experiments

Cowton, Jake; Dallmeier-Tiessen, Suenje; Fokianos, Pamfilos; Rueda, Laura; Herterich, Patricia; Kunčar, Jiří; Šimko, Tibor; Smith, T. J.

doi:10.1088/1742-6596/664/3/032030

Cited by 22 publications

(8 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data are available within the collaborations for analyses, and only the final results, usually appearing in the form of plots and tables, are included in the final publication. In 2014, after complex preparation, CERN launched its Open Data Portal to address the matter of data sharing [5]. The platform made preserved datasets and accompanying tools accessible to both the general public and the research community, yet through it emerged new challenges in research provenance preservation, which is essential for analyses reusability, reproducibilty and discoverability.…”

Section: Introductionmentioning

confidence: 99%

CERN Analysis Preservation: A Novel Digital Library Service to Enable Reusable and Reproducible Research

Chen

Dallmeier-Tiessen

Dani

et al. 2016

Research and Advanced Technology for Digital Libraries

View full text Add to dashboard Cite

Abstract. The latest policy developments require immediate action for data preservation, as well as reproducible and Open Science. To address this, an unprecedented digital library service is presented to enable the High-Energy Physics community to preserve and share their research objects (such as data, code, documentation, notes) throughout their research process. While facing the challenges of a "big data" community, the internal service builds on existing internal databases to make the process as easy and intrinsic as possible for researchers. Given the "work in progress" nature of the objects preserved, versioning is supported. It is expected that the service will not only facilitate better preservation techniques in the community, but will foremost make collaborative research easier as detailed metadata and novel retrieval functionality provide better access to ongoing works. This new type of e-infrastructure, fully integrated into the research workflow, could help in fostering Open Science practices across disciplines.

show abstract

Section: Introductionmentioning

confidence: 99%

CERN Analysis Preservation: A Novel Digital Library Service to Enable Reusable and Reproducible Research

Chen

Dallmeier-Tiessen

Dani

et al. 2016

Research and Advanced Technology for Digital Libraries

View full text Add to dashboard Cite

show abstract

“…For high-energy physics experiments, the preservation of simulated and recorded collision events (condition 3) requires sophisticated measures. Integration with LHC open-data initiatives [15] is mandatory and therefore encouraged by Law.…”

Section: Analysis Preservationmentioning

confidence: 99%

Design Pattern for Analysis Automation on Distributed Resources using Luigi Analysis Workflows

Rieger

2020

EPJ Web Conf.

View full text Add to dashboard Cite

In particle physics, workflow management systems are primarily used as tailored solutions in dedicated areas such as Monte Carlo event generation. However, physicists performing data analyses are usually required to steer their individual workflows manually, which is time-consuming and often leads to undocumented relations between particular workloads. We present the Luigi Analysis Workflows (Law) Python package, which is based on the opensource pipelining tool Luigi, originally developed by Spotify. It establishes a generic design pattern for analyses of arbitrary scale and complexity, and shifts the focus from executing to defining the analysis logic. Law provides the building blocks to seamlessly integrate interchangeable remote resources without, however, limiting itself to a specific choice of infrastructure. In particular, it encourages and enables the separation of analysis algorithms on the one hand, and run locations, storage locations, and software environments on the other hand. To cope with the sophisticated demands of end-to-end HEP analyses, Law supports job execution on WLCG infrastructure (ARC, gLite) as well as on local computing clusters (HTCondor, LSF), remote file access via most common protocols through the GFAL2 library, and an environment sandboxing mechanism with support for Docker and Singularity containers. Moreover, the novel approach ultimately aims for analysis preservation out-of-the-box. Law is entirely experiment independent and developed open-source. It is successfully used in tt̄H cross section measurements and searches for di-Higgs boson production with the CMS experiment.

show abstract

“…HEP researchers require a very specific workflow structure, with strictly defined policies on the access and preservation of their data, which all of the LHC collaborations follow. According to the Study Group for Data Preservation and Long Term Analysis in High Energy Physics, the previously mentioned policies state [5]:…”

Section: Motivationmentioning

confidence: 99%

CERN Analysis Preservation and Reuse Framework: FAIR research data services for LHC experiments

et al. 2020

View full text Add to dashboard Cite

In this paper we present the CERN Analysis Preservation service as a FAIR (Findable, Accessible, Interoperable and Reusable) research data preservation repository platform for LHC experiments. The CERN Analysis Preservation repository allows LHC collaborations to deposit and share the structured information about analyses as well as to capture the individual data assets associated to the analysis. We describe the typical data ingestion pipelines, through which an individual physicist can preserve and share their final n-tuples, ROOT macros, Jupyter notebooks, or even their full analysis workflow code and any intermediate datasets of interest for preservation within the restricted context of experimental collaboration. We discuss the importance of annotating the deposited content with high-level structured information about physics concepts in order to promote information discovery and knowledge sharing inside the collaboration. Finally, we describe techniques used to facilitate the reusability of preserved data assets by capturing and re-executing reproducible recipes and computational workflows using the REANA Reusable Analysis platform.

show abstract

Open Data and Data Analysis Preservation Services for LHC Experiments

Cited by 22 publications

References 6 publications

CERN Analysis Preservation: A Novel Digital Library Service to Enable Reusable and Reproducible Research

CERN Analysis Preservation: A Novel Digital Library Service to Enable Reusable and Reproducible Research

Design Pattern for Analysis Automation on Distributed Resources using Luigi Analysis Workflows

CERN Analysis Preservation and Reuse Framework: FAIR research data services for LHC experiments

Contact Info

Product

Resources

About