DIRAC at JINR as a general purpose system for massive computations

Abstract

Joint Institute for Nuclear Research (JINR) has several large computing facilities: Tier1 and Tier2 grid clusters, Govorun supercomputer, cloud, and LHEP computing cluster. Each of them has different access protocols, authentication and authorization procedures, data access methods. With the help of the DIRAC Interware, we were able to integrate all these resources to provide a uniform access to all these facilities. Right now, it is possible to perform basic workflows on all resources. The main use-cases covered by the DIRAC service in JINR: centralized Monte-Carlo simulation for the MPD experiment, Monte-Carlo for the Baikal-GVD neutrino telescope, as well as running jobs for the Folding@HOME project. During the pre-production stage, it is important to estimate the characteristics of user jobs. That information is crucial for planning of the execution process on different resources. An approach was elaborated to collect data about RAM, CPU, and network consumption by each user job. This helped to tune the mass production algorithms before the production actually starts. Since the system processes tens of thousands of similar jobs during one particular production, it appeared to be possible to collect from DIRAC data about these jobs and perform analysis of their execution parameters. We collect data related to the CPU model, wall-time, CPU benchmark DB12, hostname, username, and resource name. An approach was elaborated to extract meta-data about job execution and visualize it. With this visualization, it became possible to compare different computing resources, study the CPU and worker node performance. This approach does not require submitting special jobs, so the resources are not wasted for this analysis.