Creating a content delivery network for general science on the internet backbone using XCaches

Fajardo, Edgar; Weitzel, Derek; Rynge, Mats; Zvada, M.; Hicks, J. Braxton; Selmeci, Mat; Lin, Brian; Paschos, Pascal; Bockelman, Brian; Hanushevsky, Andrew; Würthwein, F.; Sfiligoi, I.

doi:10.1051/epjconf/202024504041

Cited by 10 publications

(10 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Coffea-casa users need to access data hosted by an HEP experiment; while these are traditionally secured with GSI, users do not have a GSI credential within the facility. Instead, the auto-generated data access token can be used to authenticate with a proxy service based on XRootD/XCache [31]. The proxy service, operated by the facility administrators, has a valid credential for accessing experiment data.…”

Section: Integration With Data Accessmentioning

confidence: 99%

Coffea-casa: an analysis facility prototype

et al. 2021

Self Cite

View full text Add to dashboard Cite

Data analysis in HEP has often relied on batch systems and event loops; users are given a non-interactive interface to computing resources and consider data event-by-event. The “Coffea-casa” prototype analysis facility is an effort to provide users with alternate mechanisms to access computing resources and enable new programming paradigms. Instead of the command-line interface and asynchronous batch access, a notebook-based web interface and interactive computing is provided. Instead of writing event loops, the columnbased Coffea library is used. In this paper, we describe the architectural components of the facility, the services offered to end users, and how it integrates into a larger ecosystem for data access and authentication.

show abstract

Section: Integration With Data Accessmentioning

confidence: 99%

Coffea-casa: an analysis facility prototype

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Coffea-casa users need to access data hosted by an HEP experiment; while these are traditionally secured with GSI, users do not have a GSI credential within the facility. Instead, the auto-generated data access token can be used to authenticate with an proxy service based on XRootD/XCache [29]. The proxy service, operated by the facility administrators, has a valid credential for accessing experiment data.…”

Section: Integration With Data Accessmentioning

confidence: 99%

Coffea-casa: an analysis facility prototype

Adamec,

Attebury,

Bloom

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Data analysis in HEP has often relied on batch systems and event loops; users are given a non-interactive interface to computing resources and consider data event-by-event. The "Coffea-casa" prototype analysis facility is an effort to provide users with alternate mechanisms to access computing resources and enable new programming paradigms. Instead of the command-line interface and asynchronous batch access, a notebook-based web interface and interactive computing is provided. Instead of writing event loops, the columnbased Coffea library is used. In this paper, we describe the architectural components of the facility, the services offered to end users, and how it integrates into a larger ecosystem for data access and authentication.

show abstract

“…Applications are expected to be directed to a "regional" XCache via the configuration of their runtime environment, e.g. via GeoIP as is done in the OSG Data Federation [9,17,18]. Cache misses are handled by XCache as simple XRoot-client calls to the data federation that implements the data lake.…”

Section: Xcachementioning

confidence: 99%

“…This local cache is useful in speeding up access to the few popular datasets that might be used by multiple scientists at the same location. However, the benefits may be limited [9,18] because they depend on the inadvertent intersection of multiple factors, such as scientists with the same interest, computing access, and dataset access at a particular location.…”

Section: Introductionmentioning

confidence: 99%

Analyzing Scientific Data Sharing Patterns for In-network Data Caching

Copps

Zhang

Sim

et al. 2020

Proceedings of the 2021 on Systems and Network Telemetry and Analytics

Self Cite

View full text Add to dashboard Cite

The volume of data moving through a network increases with new scientific experiments and simulations. Network bandwidth requirements also increase proportionally to deliver data within a certain time frame. We observe that a significant portion of the popular dataset is transferred multiple times to different users as well as to the same user for various reasons. In-network data caching for the shared data has shown to reduce the redundant data transfers and consequently save network traffic volume. In addition, overall application performance is expected to improve with in-network caching because access to the locally cached data results in lower latency. This paper shows how much data was shared over the study period, how much network traffic volume was consequently saved, and how much the temporary in-network caching increased the scientific application performance. It also analyzes data access patterns in applications and the impacts of caching nodes on the regional data repository. From the results, we observed that the network bandwidth demand was reduced by nearly a factor of 3 over the study period. CCS CONCEPTS• Networks → Network performance analysis; • Computing methodologies → Distributed computing methodologies.

show abstract

Creating a content delivery network for general science on the internet backbone using XCaches

Cited by 10 publications

References 5 publications

Coffea-casa: an analysis facility prototype

Coffea-casa: an analysis facility prototype

Coffea-casa: an analysis facility prototype

Analyzing Scientific Data Sharing Patterns for In-network Data Caching

Contact Info

Product

Resources

About