The DIRAC interware: current, upcoming and planned capabilities and technologies

Abstract

Efficient access to distributed computing and storage resources is mandatory for the success of current and future High Energy and Nuclear Physics Experiments. DIRAC is an interware to build and operate distributed computing systems. It provides a development framework and a rich set of services for the Workload, Data and Production Management tasks of large scientific communities. A single DIRAC installation provides a complete solution for the distributed computing of one, or more than one collaboration. The DIRAC Workload Management System (WMS) provides a transparent, uniform interface for managing computing resources. The DIRAC Data Management System (DMS) offers all the necessary tools to ensure data handling operations: it supports transparent access to storage resources based on multiple technologies, and is easily expandable. Distributed Data management can be performed, also using third party services, and operations are resilient with respect to failures. DIRAC is highly customizable and can be easily extended. For these reasons, a vast and heterogeneous set of scientific collaborations have adopted DIRAC as the base for their computing models. Users from different experiments can interact with the system in different ways, depending on their specific tasks, expertise level and previous experience using command line tools, python APIs or Web Portals. The requirements of the diverse DIRAC user communities and hosting infrastructures triggered multiple developments to improve the system usability: examples include the adoption of industry standard authorization and authentication infrastructure solutions, the management of diverse computing resources (cloud, HPC, GPGPU, etc.), the handling of high-intensity work and data flows, but also advanced monitoring and accounting using no-SQL based solutions and message queues. This contribution will highlight DIRAC’s current, upcoming and planned capabilities and technologies.

Experience with Rucio in the wider HEP community

“…The computing model of Belle II is based on Dirac [6] and more specifically on an extension of Dirac called BelleDirac that has been responsible for the Workflow Management and Data Management. BelleDirac was well suited for Belle II Data Management's need, but to tackle the massive increase of data during the next years a extensive new developments were foreseen.…”

The Cherenkov Telescope Array Observatory workflow management system

“…To handle workload and workflows for CTAO on this distributed infrastructure, a production system prototype has been developed, based on DIRAC [2] [3]. It is an interware used by many experiments to exploit distributed heterogeneous computing and storage resources.…”

DIRAC at JINR as a general purpose system for massive computations

“…DIRAC Interware is an open-source development platform for the integration of heterogeneous computing and storage resources [1]. It provides a central task queue and basic tools for data management alongside with accounting system, user management system, workflow management system, and other systems.…”