Hybrid Distributed Computing Service Based on the DIRAC Interware

Gergel, Victor; Korenkov, V.; Pelevanyuk, I.; Sapunov, M.; Tsaregorodtsev, A.; Zrelov, P.

doi:10.1007/978-3-319-57135-5_8

Cited by 4 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It was used for educational purposes only to demonstrate the concept of distributed high-throughput computing for students. dCache became the first storage system integrated, and Tier2 cluster was the first integrated computing element [2].…”

Section: Dirac Development In Jinrmentioning

confidence: 99%

DIRAC Interware as a Service for High-Thoughput Computing in JINR

Pelevanyuk

Tsaregorodtsev

2022

Phys. Part. Nuclei Lett.

View full text Add to dashboard Cite

DIRACInterware is an open-source development platform for the integration of heterogeneous computing and storage resources. The service based on this platform was deployed and configured in Joint Institute for Nuclear Research in 2016. Now it is actively used for MPD, Baikal-GVD, and BM@N experiments. In JINR we have five big computing resources with uniform access via the DIRAC service: Tier1, Tier2, Govorun supercomputer, cloud, and NICA cluster. In particular, the DIRAC service was used as a tool for the integration of cloud resources of JINR member states. The overall performance of the united system is at least three times more efficient compared to the use of any single computing resource. Of course, using the united system adds complexity for users and requires additional effort to reach high performance. But, for the last three years of active use of the DIRAC, the approaches were elaborated to simplify the use of the system. Right now there are many tools and components developed to allow the fast construction of new workflows. The total number of completed jobs exceeds 1 million, and the total amount of computing work is around 4.5 million HS06days.

show abstract

Section: Dirac Development In Jinrmentioning

confidence: 99%

DIRAC Interware as a Service for High-Thoughput Computing in JINR

Pelevanyuk

Tsaregorodtsev

2022

Phys. Part. Nuclei Lett.

View full text Add to dashboard Cite

show abstract

“…The agent-based WMS increases the efficiency of user payloads execution in a distributed system due to a preliminary validation of the execution environment reducing considerably execution failure rates. The agent-based WMS allows also to abstract away differences in access protocols of heterogeneous computing resources providers and aggregate them as a set of similar logical computing clusters, thus solving the problem of the resource's heterogeneity [9].…”

Section: Dirac Interwarementioning

confidence: 99%

Integration of the JINR Hybrid Computing Resources with the DIRAC Interware for Data Intensive Applications

Korenkov

Pelevanyuk

Tsaregorodtsev

2020

Communications in Computer and Information Science

Self Cite

View full text Add to dashboard Cite

Scientific data and computing-intensive applications become more and more widely used. Different computing solutions have different protocols and architectures, they should be chosen carefully in the design of computing projects of large scientific communities. In a modern world of diverse computing resources such as grids, clouds, and supercomputers the choice can be difficult. Therefore, software developed for integration of various computing and storage resources into a single infrastructure, the so-called interware, makes this choice easier. The DIRAC interware is one of these products. It proved to be an effective solution for many experiments in High Energy Physics and some other areas of science providing means for seamless access to distributed computing and storage resources. The DIRAC interware was deployed in the Joint Institute for Nuclear Research to serve the needs of different scientific groups by providing a single interface to a variety of computing resources: grid cluster, computing cloud, supercomputer Govorun, disk and tape storage systems. The DIRAC based solution was proposed for the currently operational Baryonic Matter at Nuclotron experiment as well as for the future experiment Multi-Purpose Detector at the Nuclotron-based Ion Collider fAcility. Both experiments have requirements making the use of heterogeneous computing resources necessary. A set of tests was introduced in order to demonstrate the performance of the JINR distributed computing system.

show abstract

“…The standard incremental process of system enhancement went through three basic steps related to: DIRAC setup and configuration; integration of various computing and storage resources; and elaboration of approaches for effective use of resources. Minimally working infrastructure was ready by 2017 [3].…”

Section: Introducing Dirac In Jinrmentioning

confidence: 99%

DIRAC at JINR as a general purpose system for massive computations

Korenkov

Pelevanyuk²,

Tsaregorodtsev³

2023

J. Phys.: Conf. Ser.

Self Cite

View full text Add to dashboard Cite

Joint Institute for Nuclear Research (JINR) has several large computing facilities: Tier1 and Tier2 grid clusters, Govorun supercomputer, cloud, and LHEP computing cluster. Each of them has different access protocols, authentication and authorization procedures, data access methods. With the help of the DIRAC Interware, we were able to integrate all these resources to provide a uniform access to all these facilities. Right now, it is possible to perform basic workflows on all resources. The main use-cases covered by the DIRAC service in JINR: centralized Monte-Carlo simulation for the MPD experiment, Monte-Carlo for the Baikal-GVD neutrino telescope, as well as running jobs for the Folding@HOME project. During the pre-production stage, it is important to estimate the characteristics of user jobs. That information is crucial for planning of the execution process on different resources. An approach was elaborated to collect data about RAM, CPU, and network consumption by each user job. This helped to tune the mass production algorithms before the production actually starts. Since the system processes tens of thousands of similar jobs during one particular production, it appeared to be possible to collect from DIRAC data about these jobs and perform analysis of their execution parameters. We collect data related to the CPU model, wall-time, CPU benchmark DB12, hostname, username, and resource name. An approach was elaborated to extract meta-data about job execution and visualize it. With this visualization, it became possible to compare different computing resources, study the CPU and worker node performance. This approach does not require submitting special jobs, so the resources are not wasted for this analysis.

show abstract

Hybrid Distributed Computing Service Based on the DIRAC Interware

Cited by 4 publications

References 12 publications

DIRAC Interware as a Service for High-Thoughput Computing in JINR

DIRAC Interware as a Service for High-Thoughput Computing in JINR

Integration of the JINR Hybrid Computing Resources with the DIRAC Interware for Data Intensive Applications

DIRAC at JINR as a general purpose system for massive computations

Contact Info

Product

Resources

About