A federated Xrootd cache

Fajardo, Edgar; Tadel, Alja Mrak; Tadel, M.; Steer, B.; Martin, Terrence; Würthwein, F.

doi:10.1088/1742-6596/1085/3/032025

Cited by 14 publications

(20 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• SoCal working set: only considers access at the UCSD and Caltech sites. It is calculated using the XRootD monitoring data of the SoCal cache [5].…”

Section: Data Tiermentioning

confidence: 99%

“…The cache fetches files from the federation, storing them locally. For the CMS LHC-Run-II, UCSD scale tested and deployed a multi node federated XCache (i.e federated implies the cache is presented as one logical entrypoint to the client, but it is made out of several servers) that could serve a limited part of the analysis namespace [5]. We refer to this deployment as the SoCal cache.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Moving the California distributed CMS XCache from bare metal into containers using Kubernetes

et al. 2020

View full text Add to dashboard Cite

The University of California system maintains excellent networking between its campuses and a number of other Universities in California, including Caltech, most of them being connected at 100 Gbps. UCSD and Caltech Tier2 centers have joined their disk systems into a single logical caching system, with worker nodes from both sites accessing data from disks at either site. This successful setup has been in place for the last two years. However, coherently managing nodes at multiple physical locations is not trivial and requires an update on the operations model used. The Pacific Research Platform (PRP) provides Kubernetes resource pool spanning resources in the science demilitarized zones (DMZs) in several campuses in California and worldwide. We show how we migrated the XCache services from bare-metal deployments into containers using the PRP cluster. This paper presents the reasoning behind our hardware decisions and the experience in migrating to and operating in a mixed environment.

show abstract

“…• SoCal working set: only considers access at the UCSD and Caltech sites. It is calculated using the XRootD monitoring data of the SoCal cache [5].…”

Section: Data Tiermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Moving the California distributed CMS XCache from bare metal into containers using Kubernetes

et al. 2020

View full text Add to dashboard Cite

show abstract

“…It comes with no surprise that XRootD development is largely driven by the use cases coming from the WLCG project, as it is the backbone of numerous software defined storage solutions (like EOS [2] and DPM [3]) used to accommodate the vast amount of data registered by the LHC experiments at CERN, most notably Atlas [4], CMS [5], LHCb [6] and Alice [7]. One of the key components of the XRootD framework is the C++ client, which is fundamental not only to the command line utilities like xrdcp and xrdfs, but also to XCache (a XRootD file-based caching proxy) [8] [9], XrootdFS (a FUSE based mountable file system) [10] and EOS (the storage service of choice at CERN). In addition, the XRootD client is employed to provide remote data access in many physics analysis frameworks like ROOT [11] and in data movers like FTS [12].…”

Section: Introductionmentioning

confidence: 99%

Exploring the virtues of XRootD5: Declarative API

Simon

Hanushevsky²

2021

EPJ Web Conf.

View full text Add to dashboard Cite

Across the years, being the backbone of numerous data management solutions used within the WLCG collaboration, the XRootD framework and protocol became one of the most important building blocks for storage solutions in the High Energy Physics (HEP) community. The latest big milestone for the project, release 5, introduced multitude of architectural improvements and functional enhancements, including the new client side declarative API, which is the main focus of this study. In this contribution, we give an overview of the new client API and we discuss its motivation and its positive impact on overall software quality (coupling, cohesion), readability and composability.

show abstract

“…For this study, we collected data access measurements from the Southern California Petabyte Scale Cache [8], where client jobs requested data files for High-Luminosity Large Hadron Collider (HL-LHC) analysis. We studied how much data is shared, how much network traffic volume is consequently saved, and how much the innetwork data cache increases application performance.…”

Section: Introductionmentioning

confidence: 99%

Analyzing Scientific Data Sharing Patterns for In-network Data Caching

Copps

Zhang

Sim

et al. 2020

Proceedings of the 2021 on Systems and Network Telemetry and Analytics

View full text Add to dashboard Cite

The volume of data moving through a network increases with new scientific experiments and simulations. Network bandwidth requirements also increase proportionally to deliver data within a certain time frame. We observe that a significant portion of the popular dataset is transferred multiple times to different users as well as to the same user for various reasons. In-network data caching for the shared data has shown to reduce the redundant data transfers and consequently save network traffic volume. In addition, overall application performance is expected to improve with in-network caching because access to the locally cached data results in lower latency. This paper shows how much data was shared over the study period, how much network traffic volume was consequently saved, and how much the temporary in-network caching increased the scientific application performance. It also analyzes data access patterns in applications and the impacts of caching nodes on the regional data repository. From the results, we observed that the network bandwidth demand was reduced by nearly a factor of 3 over the study period. CCS CONCEPTS• Networks → Network performance analysis; • Computing methodologies → Distributed computing methodologies.

show abstract

A federated Xrootd cache

Cited by 14 publications

References 2 publications

Moving the California distributed CMS XCache from bare metal into containers using Kubernetes

Moving the California distributed CMS XCache from bare metal into containers using Kubernetes

Exploring the virtues of XRootD5: Declarative API

Analyzing Scientific Data Sharing Patterns for In-network Data Caching

Contact Info

Product

Resources

About