Dynamic federations: storage aggregation using open tools and protocols

Abstract

A number of storage elements now offer standard protocol interfaces like NFS 4.1/pNFS and WebDAV, for access to their data repositories, in line with the standardization effort of the European Middleware Initiative (EMI). Also the LCG FileCatalogue (LFC) can offer such features. Here we report on work that seeks to exploit the federation potential of these protocols and build a system that offers a unique view of the storage and metadata ensemble and the possibility of integration of other compatible resources such as those from cloud providers. The challenge, here undertaken by the providers of dCache and DPM, and pragmatically open to other Grid and Cloud storage solutions, is to build such a system while being able to accommodate name translations from existing catalogues (e.g. LFCs), experimentbased metadata catalogues, or stateless algorithmic name translations, also known as "trivial file catalogues". Such so-called storage federations of standard protocols-based storage elements give a unique view of their content, thus promoting simplicity in accessing the data they contain and offering new possibilities for resilience and data placement strategies. The goal is to consider HTTP and NFS4.1-based storage elements and metadata catalogues and make them able to cooperate through an architecture that properly feeds the redirection mechanisms that they are based upon, thus giving the functionalities of a "loosely coupled" storage federation. One of the key requirements is to use standard clients (provided by OS'es or open source distributions, e.g. Web browsers) to access an already aggregated system; this approach is quite different from aggregating the repositories at the client side through some wrapper API, like for instance GFAL, or by developing new custom clients. Other technical challenges that will determine the success of this initiative include performance, latency and scalability, and the ability to create worldwide storage federations that are able to redirect clients to repositories that they can efficiently access, for instance trying to choose the endpoints that are closer or applying other criteria. We believe that the features of a loosely coupled federation of open-protocols-based storage elements will open many possibilities of evolving the current computing models without disrupting them, and, at the same time, will be able to operate with the existing infrastructures, follow their evolution path and add storage centers that can be acquired as a third-party service.

Distributed cloud computing in high energy physics

“…If multiple copies of the data reside at different sites, then the selected site would be determined by the geographical location of the client with respect to the storage servers. We are testing a new federated WebDAV system that meets our design requirements [29]. WebDAV is a widely adopted open standard based on HTTP protocols [28].…”

Efficient HTTP Based I/O on Very Large Datasets for High Performance Computing with the Libdavix Library

“…The combined usage of libdavix for data analysis with a Replica catalogue or federation system supporting able to provide Metalink files like the DynaFed system (Dynamic Storage Federation) [19] enforces the global resilience of the I/O layer of any HPC application in a transparent manner. It provides the guarantee that a read operation on a resource will succeed as long as one replica of this resource is remotely accessible and referenced by the corresponding Metalink.…”

HEP computing in a context-aware cloud environment

“…We are testing a new federated WebDAV system that meets our design requirements [5]. There is an alternative solution being developed internally by the ATLAS project called FAX [15].…”