The Dynamic Federations: federate Storage on the fly using HTTP/WebDAV and DMLite

Abstract

Recently the importance of clustering storage nodes across site boundaries is becoming more clear, thanks also to the recent ongoing initiatives in the CMS and ATLAS experiments. These approaches are supposed to promote simplicity in accessing the data and offering new possibilities for resilience and data placement strategies, that may also lead to a better utilization of the available CPU slots. Here we report on work that seeks to exploit the federation potential of redirectable protocols like HTTP/WebDAV and build a dynamic, scalable, persistency-free system that offers a unique view of the storage and metadata ensemble and the possibility of integration of other compatible resources such as those from cloud providers. The challenge, here undertaken by the providers of dCache and DPM, partially in the context of EMI-data, and pragmatically open to any other Grid and Cloud storage solutions, is to build such a system while being able to accommodate name translations from existing catalogues (e.g. LFCs), experiment-based metadata catalogues, or stateless algorithmic name translations, also known as "trivial file catalogues". Other technical challenges that will determine the success of this initiative include performance, latency and scalability, and the ability to create worldwide storage federations that are able to redirect clients to repositories that they can efficiently access, for instance trying to choose the endpoints that are closer or applying other criteria. One of the key requirements is to use standard clients (provided by OS'es or open source distributions, e.g. Web browsers) to access an already aggregated system. This was accomplished by exploiting the possibilities offered by the DMLite framework, in order to integrate with existing frontends like the Apache server. We believe that the features of a loosely coupled federation of open-protocols-based storage elements will open many possibilities of evolving the current computing models without disrupting them, and, at the same time, can operate with the existing infrastructures, follow their evolution path and add storage centers that can be acquired as a third-party service.

CDFS: A high-efficiency Data Access System for Storage Federations

“…At the first mode, the system has its name services providing global file namespace, such as Rucio [2]. The second mode can automatically aggregate the global files view based on broadcast, such as Dynafed [3]. Although its speed of file-searching is slower than the first mode, it is more simple and dynamic.…”

An http data-federation ecosystem with caching functionality using DPM and Dynafed

“…Dynafed, as a new tool for data federation has been presented in [1] and its usage is became quickly very popular in the contest of WLCG communities [6]. In addition we quote the literature related to the investigation for the deployment of new caching systems and their relative integration in applicative workflows, some works has been presented in the context of different experiment like ATLAS and CMS [6] [7].…”

“…Today all most used storage technologies provide such kind of support, allowing to exploit a set of new setup, even thanks to the availability of new tools and software created to take advantage of the specific features of HTTP. Among them, Dynafed [1] developed at CERN, represents one of the emerging technologies to easily federate distributed endpoints presenting them as a single namespace. The mount point is accessible via HTTP/WebDAV, offering at the same time a very effective user-experience.…”

INDIGO-Datacloud: foundations and architectural description of a Platform as a Service oriented to scientific computing

“…Furthermore, all the standard interfaces exposed by the data management components will be accessible to users' applications as well through standard protocols such as CDMI [33] and WebDAV [34].…”