Improvement of AMGA Python Client Library for Belle II Experiment

Kwak, Jae-Hyuck; Park, Geunchul; Huh, Taesang; Hwang, Soonwook

doi:10.1088/1742-6596/664/4/042041

Cited by 6 publications

(4 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1. There are four principal components in the diagram: Sites, the DIRAC Interware [5], its extension for Belle II named as BelleDIRAC [4,6,7] and grid services which are third party components such as the LCG File Catalog (LFC) [8], AMGA [9][10][11] and FTS [12]. Sites provide two basic resources: Computing Elements (CE) and Storage Elements (SE).…”

Section: Current Distributed Data Management Systemmentioning

confidence: 99%

Distributed data management on Belle II

et al. 2020

View full text Add to dashboard Cite

The Belle II experiment started taking physics data in April 2018 with an estimated total volume of all files including raw events, Monte-Carlo and skim statistics of 340 petabytes expected by the end of operations in the late-2020s. Originally designed as a fully integrated component of the BelleDIRAC production system, the Belle II distributed data management (DDM) software needs to manage data across about 29 storage elements worldwide for a collaboration of nearly 1000 physicists. By late 2018, this software required significant performance improvements to meet the requirements of physics data taking and was seriously lacking in automation. Rucio, the DDM solution created by ATLAS, was an obvious alternative but required tight integration with BelleDIRAC and a seamless yet non-trivial migration. This contribution describes the work done on both DDM options, the current status of the software running successfully in production and the problems associated with trying to balance long-term operations cost against short term risk.

show abstract

Section: Current Distributed Data Management Systemmentioning

confidence: 99%

Distributed data management on Belle II

et al. 2020

View full text Add to dashboard Cite

show abstract

“…An agent confirms the accessibility of the file, extracts the metadata embedded on the files, and groups them into data sets based on the corresponding experiment number, run type, and run number. The file is uploaded into a temporal storage element at the host laboratory (KEK) and its information is registered into the replica catalog LFC [6] and the metadata catalog AMGA [7][8][9]. Once the process is completed, replication to permanent raw data centers is triggered using the Distributed Data Management system of Belle II [10].…”

Section: Raw Data Registration and Replicationmentioning

confidence: 99%

The Belle II Raw Data Management System

Villanueva

Ueda²

2020

EPJ Web Conf.

View full text Add to dashboard Cite

The Belle II experiment, a major upgrade of the previous e+e− asymmetric collider experiment Belle, is expected to produce tens of petabytes of data per year due to the luminosity increase from the upgraded SuperKEKB accelerator. The distributed computing system of the Belle II experiment plays a key role, storing and distributing data in a reliable way to be easily accessed and analyzed by more than 1000 collaborators. In particular, the Belle II Raw Data Management system has been developed with an aim to upload output files onto grid storage, register them into the file and metadata catalogs, and make two replicas of the full raw data set using the Belle II Distributed Data Management system. It has been implemented as an extension of DIRAC (Distributed Infrastructure with Remote Agent Control) and consists of a database, services, client and monitoring tools, and several agents that treat the data automatically. The first year of data taken with the Belle II full detector has been managed by the Belle II Raw Data Management system successfully. The design, current status, and performance are presented. Prospects for improvements towards the full luminosity data taking are also reviewed.

show abstract

“…At the beginning, only a limited number of jobs can be handled due to low performance of AMGA or failure of job when downloading (uploading) input (output) data. We have gradually improved the sytem to obtain the higher throughput [7,8] and number of jobs have been increased. In addition, we have been developing the production system in order to automate the MC production.…”

Section: Monte-carlo Productions Campaignsmentioning

confidence: 99%

Overview of the Belle II Computing

Kato¹

2017

Proceedings of the 3rd International Symposium on “Quest for the Origin of Particles and the Universe" — PoS(KMI2017)

View full text Add to dashboard Cite

In the Belle II experiment, a data sample of 50 ab −1 will be collected. The Belle II computing system is expected to manage the processing of massive raw data, production of copious simulation as well as many concurrent user analysis jobs. We adopted the distributed computing model to realize it. In this contribution, the overview and highlights of the Belle II computing system and activity of KMI are presented.

show abstract

Improvement of AMGA Python Client Library for Belle II Experiment

Cited by 6 publications

References 2 publications

Distributed data management on Belle II

Distributed data management on Belle II

The Belle II Raw Data Management System

Overview of the Belle II Computing

Contact Info

Product

Resources

About