Managing ATLAS data on a petabyte-scale with DQ2

The Present and Future Challenges of Distributed Computing in the ATLAS experiment

“…In the ATLAS Computing Model, each Tier-1 centre has a group of Tier-2 centres associated to it and the data distribution over the Grid is managed in an organised way with this association. The distributed data management system (DDM/DQ2) [6] was first implemented in a way that data transfers between Tier-2 centres associated to different Tier-1 centres were made via the Tier-1 centres. Based on operational experiences and the measured transfer rates, DDM has evolved, moving from a "tree" topology to a "mesh" of sites of any Tiers, to enable direct transfers when estimated transfer times are shorter based on the measured statistics in the recent past, for efficiency, less load to the system, and end-user convenience [7].…”

“…Rucio is the next ATLAS distributed data management system [6], which enables global management of the space, rather than per-site. It is being developed utilising not only the relational database (Oracle) used in the current system, but also a non-relational structured storage (Hadoop).…”

ATLAS Operations: Experience and Evolution in the Data Taking Era

“…The data on Tier-2 centres are replicated from their associated Tier-1 centres. The data to be distributed on the Grid are registered to the ATLAS distributed data management system (DDM) [6].…”

A data transfer framework for large-scale science experiments

“…PhEDEx consists of a set of agents responsible for file replication, routing decisions, tape migrations, and so on. ATLAS DDM [26] is a file-based data management system operates on the worldwide LHC computing grid infrastructure (WLCG). It is capable of managing data on the order of petabytes.…”