The ATLAS Data Carousel Project Status

Abstract

The High Luminosity upgrade to the LHC, which aims for a tenfold increase in the luminosity of proton-proton collisions at an energy of 14 TeV, is expected to start operation in 2028/29 and will deliver an unprecedented volume of scientific data at the multi-exabyte scale. This amount of data has to be stored, and the corresponding storage system must ensure fast and reliable data delivery for processing by scientific groups distributed all over the world. The present LHC computing and data management model will not be able to provide the required infrastructure growth, even taking into account the expected hardware technology evolution. To address this challenge, the Data Carousel R&D project was launched by the ATLAS experiment in the fall of 2018. State-of-the-art data and workflow management technologies are under active development, and their current status is presented here.

Updates to the ATLAS Data Carousel Project

“…The High Luminosity upgrade to the LHC [1] will deliver an unprecedented volume of scientific data at the exabyte scale. To tackle this data storage challenge, the ATLAS experiment [2] started the Data Carousel R&D project [3] since the fall of 2018. Data Carousel is a tape-driven workflow that allows jobs to get input data directly from tape by orchestrating the workflow management systems ProdSys2 [4] and PanDA [5], the distributed data management (DDM) system Rucio [6], and the tape services.…”

ATLAS Distributed Computing Evolution: Developments and Demonstrators Towards HL–LHC

“…The evolution of tape storage within ATLAS has shifted from its conventional role as pure archive storage towards a more dynamic and integrated approach. A prime example of this transformation is the successful implementation of the Data Carousel [17] mechanism in AT-LAS production operations, a strategy that has been effectively in practice for several years. At the heart of the current ATLAS R&D tape activities is the concept of smart archiving, which serves as a central focus for research and development in our tape storage strategy.…”

BigPanDA monitoring system evolution in the ATLAS Experiment

“…Initially, the project comprised a set of core views covering the basic needs of monitoring key aspects of the PanDA WMS, in particular jobs, tasks, files, datasets, computing sites, and users. With the advancement of the PanDA WMS [1] in the ATLAS Experiment, several new components have been developed, such as Harvester [10], iDDS [4], Data Carousel [5], and Global Shares [6]. All of them are closely related to key WMS objects and have to be monitored in an integrated way.…”

“…The Data Carousel [5] orchestrates data processing between workload management, data management, and storage services with the bulk data resident on tape. The processing is executed by staging and promptly processing a sliding window of inputs onto faster buffer storage such that only a small percentage of input data are available at any one time.…”

“…Since then, it has been continuously improved in line with the advancement of the PanDA WMS to meet new challenges brought by the ATLAS collaboration, in particular the integration of HPCs and commercial clouds, the central control of computing activity distributed among available resources, and orchestration of the data movement between tape storage and disks. To complete them, the several new components were developed, such as Harvester, Intelligent Data Delivery Service (iDDS) [4], Data Carousel [5], and Global Shares [6]. The overview of the ATLAS Workflow Management System is shown in Figure 1.…”