Architecture and prototype of a WLCG data lake for HL-LHC

Abstract

The computing strategy document for HL-LHC identifies storage as one of the main WLCG challenges in one decade from now. In the naive assumption of applying today's computing model, the ATLAS and CMS experiments will need one order of magnitude more storage resources than what could be realistically provided by the funding agencies at the same cost of today. The evolution of the computing facilities and the way storage will be organized and consolidated will play a key role in how this possible shortage of resources will be addressed. In this contribution we will describe the architecture of a WLCG data lake, intended as a storage service geographically distributed across large data centers connected by fast network with low latency. Will present the experience with our first prototype, showing how the concept, implemented at different scales, can serve different needs, from regional and national consolidation of storage to an international data provisioning service. We will highlight how the system leverages its distributed nature, the economy of scale and different classes of storage to optimise the hardware and operational cost, through a set of policy driven decisions concerning data placement and data retention. We will discuss how the system leverages or interoperates with existing federated storage solutions. We will finally describe the possible data processing models in this environment and present our first benchmarks.

Network Capabilities for the HL-LHC Era

“…In addition, the usual approaches that work for configuration of compute might not work for network infrastructure due to various reasons. • Data Center Interconnect technologies (both HW and software-based) are quite novel approaches that will require initial deployments to evaluate how they could benefit inter-DC networking for federated use case such as data lakes [8]. • A range of other approaches that are only mentioned briefly in the report such as GPU, virtualized storage, hyper-converged architectures and edge services for HEP instrumentation and experiments will require initial testbeds and evaluations.…”

“…A cluster of such centres can then create a federated site that will be exposed behind a single endpoint/interface for the experiments. This transformation has already started within the WLCG data lakes activities and some of the participating sites are already running their storage and or compute in a federated setup [8].…”

dCache - Keeping up With the Evolution of Science

“…Though AGLT2 and NDGF sites operate over a decade, such distributed setups are exceptions. However, to reduce operational overhead, more and more small Tier-2s will come together to build so-called data lake [7].…”

Evolution of HammerCloud to commission CERN Compute resources

“…from supporting only purely CPU-bound workflows, we are now able to support a variety of workflows with emphasis on different type of resource and different level of CPU-, memory-, and I/O-intensity. We can leverage an enriched variety of workflows to setup a testbed for WLCG Data Organization, Management, and Access [10] performance studies, to test ideas for the HL-LHC [11].…”