Advancements in data management services for distributed e-infrastructures: the eXtreme-DataCloud project

Abstract

The development of data management services capable to cope with very large data resources is a key challenge to allow the future einfrastructures to address the needs of the next generation extreme scale scientific experiments. To face this challenge, in November 2017 the H2020 eXtreme DataCloud -XDC project has been launched. Lasting for 27 months and combining the expertise of eight large European research organisations, the project aims at developing scalable technologies for federating storage resources and managing data in highly distributed computing environments. The targeted platforms are the current and next generation e-Infrastructures deployed in Europe, such as the European Open Science Cloud (EOSC), the European Grid Infrastructure (EGI), and the Worldwide LHC Computing Grid (WLCG). The project is use-case driven with a multidisciplinary approach, addressing requirements from research communities belonging to a wide range of scientific domains: High Energy Physics, Astronomy, Photon and Life Science, Medical research. XDC is aimed at implementing data management scalable services, combining already established data management and orchestration tools, to address the following high level topics: policy driven data management based on Quality-of-Service, Data Life-cycle management, smart placement of data with caching mechanisms to reduce access latency, meta-data with no predefined schema handling, execution of pre-processing applications during ingestion, data management and protection of sensitive data in distributed e-infrastructures, intelligent data placement based on access patterns. This contribution introduces the project, presents the foreseen overall architecture and the developments that are being carried on to implement the requested functionalities.EPJ Web of Conferences 214, 04044 (2019)

The eXtreme-DataCloud project solutions for data management services in distributed e-infrastructures

“…After an initial phase of requirement gathering which involved all the scientific communities represented in XDC, the project produced two major releases: XDC-1, codenamed Pulsar, released in January 2018 and XDC-2, codenamed Quasar, released in March 2020. Both releases are based on the general architecture that is presented in [4] and improve or add missing functionalities to the set of software components reported in Table2, addressing important topics like federation of storage resources, smart caching solutions, policy driven data management based on Quality of Service, data lifecycle management, metadata handling and manipulation, optimized data management based on storage events. The two XDC releases allow a tighter integration between the participating components in order to enable workflows which exploit the whole data platform as a coherent infrastructure rather than a set of disparate services.…”

“…XDC developments enrich already existing services for data management in e-infrastructures adding new functionalities as requested by the research communities represented in the consortium and targeting the computing infrastructure built by European Open Science Cloud (EOSC) [2], the EGI Fundation (EGI) [3], the Worldwide LHC Computing Grid (WLCG). An introduction to the project, the description of its objectives, the involved research communities, the services composing its software catalogue and the foreseen technical architecture can be found in [4] [5]. The next section provides a brief project overview, while Section 3 reports the main development achievements of the projects.…”

CMS Software and Offline preparation for future runs

“…In the context of WLCG, a general effort towards a new-generation data infrastructure is strongly active, also via groups like DOMA[23] and European Projects like ESCAPE[24] and XDC [25]. The "data-lake" approach aims to create a solid, secure and curated data infrastructure, using HEP-owned data centers, linked via middleware allowing them to appear as much as possible as single logical system.…”