DIRAC data management: consistency, integrity and coherence of data

Bargiotti, M.; Smith, Andrew C.

doi:10.1088/1742-6596/119/6/062013

Cited by 6 publications

(3 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using the API classes it is easy to write small scripts or applications to manage user jobs [3] and data processing with metadata [6] and provenance [7] management for HTC. These scripts can be used in the client side as PaaS or encapsulated in the server side, for example in a VRE.…”

Section: Dirac Python Apimentioning

confidence: 99%

Powering Distributed Applications with DIRAC Engine

Mendez¹,

Ramo²,

Diaz³

et al. 2014

Proceedings of International Symposium on Grids and Clouds (ISGC) 2014 — PoS(ISGC2014)

View full text Add to dashboard Cite

DIRAC has evolved from a framework for distributed computing matters operated by the final user, to a platform that can also be used from third-party Virtual Research Environments (VRE), Science Gateways (SG) or distributed computing applications. Therefore, DIRAC has become a platform providing distributed computing interoperability and high level services for science. For this purpose, DIRAC supports several Application Programming Interfaces (API) designed for a versatile programming model in different developing scenarios. This paper presents such scenarios categorized by the location of the API client. Thus, a first scenario is an application server powered by DIRAC services. Second, a web browser running embedded DIRAC calls, for dynamic web client composition. Third scenario is a DIRAC Platform as a Service (PaaS) powering distributed applications with High Throughput Computing (HTC) on the client side. This paper describes the underlying DIRAC architecture and the main functionalities of each API and developing scenario. Overall DIRAC is a coherent toolkit that enables a wide range of component developments like service for scientific applications, web plugs or PaaS agents.

show abstract

Section: Dirac Python Apimentioning

confidence: 99%

Powering Distributed Applications with DIRAC Engine

Mendez¹,

Ramo²,

Diaz³

et al. 2014

Proceedings of International Symposium on Grids and Clouds (ISGC) 2014 — PoS(ISGC2014)

View full text Add to dashboard Cite

show abstract

“…Since the storage systems and catalogs are completely decoupled, these checks are not trivial. DIRAC provides several ways to carry out the checks [16]. The corresponding tools are centred around the Integrity Database which accumulates reports of inconsistencies of the contents of the storage and catalogs as well as reports on the data access problems.…”

Section: Data Management Systemmentioning

confidence: 99%

DIRAC: a community grid solution

Tsaregorodtsev

Bargiotti

Brook

et al. 2008

J. Phys.: Conf. Ser.

Self Cite

123

View full text Add to dashboard Cite

Abstract. The DIRAC system was developed in order to provide a complete solution for using the distributed computing resources of the LHCb experiment at CERN for data production and analysis. It allows a concurrent use of over 10K CPUs and 10M file replicas distributed over many tens of sites. The sites can be part of a Computing Grid such as WLCG or standalone computing clusters all integrated in a single management structure. DIRAC is a generic system with the LHCb specific functionality incorporated through a number of plug-in modules. It can be easily adapted to the needs of other communities. Special attention is paid to the resilience of the DIRAC components to allow an efficient use of non-reliable resources. The DIRAC production management components provide a framework for building highly automated data production systems including data distribution and data driven workload scheduling. In this paper we give an overview of the DIRAC system architecture and design choices. We show how different components are put together to compose an integrated data processing system including all the aspects of the LHCb experiment -from the MC production and raw data reconstruction to the final user analysis.

show abstract

“…The first distributed computing system, based on DIRAC [3] (Distributed Infrastructure with Remote Agent Control), was studied, developed, and deployed as the IHEPDIRAC project, becoming a production service for the BESIII experiment in 2012 [4]. Serving as a generic grid solution, DIRAC provides a Data Management System [5] (DMS) suitable for small sites and those lacking in grid computing maintainers. Currently, DIRAC-DMS primarily serves the JUNO experiment [6], a neutrino observatory located in southern China, which is planning to generate 2.4 PB of raw data and 0.6 PB of MC and calibration data every year [7].…”

Section: Introductionmentioning

confidence: 99%

Distributed Data Management System at IHEP

Zhang,

et al. 2023

Proceedings of International Symposium on Grids &Amp; Clouds (ISGC) 2023 in Conjunction With HEPiX Spring 2023 Workshop — PoS(I

View full text Add to dashboard Cite

A distributed grid data management system has been established at the Institute of High Energy Physics (IHEP) since 2014, leveraging the DIRAC data management system. This system has been effectively utilized by the BESIII, JUNO, and CEPC experiments. To address the diverse data scales and intricate data management requirements associated with data production, IHEP developers have embarked on endeavors to create a more adaptable and experiment-oriented grid data management system based on Rucio, owing to its proven reliability, scalability, and automation capabilities. As a result, a production system for Monte Carlo (MC) data production has been implemented, relying on the DIRAC Data Management System (DMS) solution. Furthermore, work is underway to develop Rucio data transfer daemon plugins to facilitate non-grid data transfer for upcoming experiments at IHEP. Supporting infrastructure for distributed data management has also been established, encompassing StoRM and EOS storage elements, an IAM authentication service, as well as a third-party-copy active probing system.

show abstract

DIRAC data management: consistency, integrity and coherence of data

Cited by 6 publications

References 4 publications

Powering Distributed Applications with DIRAC Engine

Powering Distributed Applications with DIRAC Engine

DIRAC: a community grid solution

Distributed Data Management System at IHEP

Contact Info

Product

Resources

About