Any Data, Any Time, Anywhere: Global Data Access for Science

Bloom, K.; Bockelman, Brian; Bradley, D.; Dasu, S.; Dost, J M; Fanzago, F.; Sfiligoi, I.; Tadel, Alja Mrak; Tadel, M.; Vuosalo, C.; Würthwein, F.; Yagil, Avi; Zvada, M.

doi:10.1109/bdc.2015.33

Cited by 19 publications

(13 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proxy service, operated by the facility administrators, has a valid credential for accessing experiment data. Data is downloaded on demand from an infrastructure like CMS's "Any Data, Any Time, Anywhere" (AAA) data federation [32]. In the current deployment, because we can guarantee all authenticated users are valid CMS members, we assume any authenticated access to the proxy service is allowed to access CMS data.…”

Section: Integration With Data Accessmentioning

confidence: 99%

Coffea-casa: an analysis facility prototype

et al. 2021

Self Cite

View full text Add to dashboard Cite

Data analysis in HEP has often relied on batch systems and event loops; users are given a non-interactive interface to computing resources and consider data event-by-event. The “Coffea-casa” prototype analysis facility is an effort to provide users with alternate mechanisms to access computing resources and enable new programming paradigms. Instead of the command-line interface and asynchronous batch access, a notebook-based web interface and interactive computing is provided. Instead of writing event loops, the columnbased Coffea library is used. In this paper, we describe the architectural components of the facility, the services offered to end users, and how it integrates into a larger ecosystem for data access and authentication.

show abstract

Section: Integration With Data Accessmentioning

confidence: 99%

Coffea-casa: an analysis facility prototype

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Due to the increasing reliability and capacity of wide area network links CMS adapted its computing model for Run-2 to allow for remote data access, which was basically excluded before. CMS commissioned a global data federation [30] that comprises all Grid Storage Elements (SE). It is sufficient to just know the logical file name (LFN) and the URL of an entry point to access any CMS file that is presently hosted on disk storage.…”

Section: Cms Configurationmentioning

confidence: 99%

Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers

Cameron

Cardo

Conciatore

et al. 2021

Comput Softw Big Sci

View full text Add to dashboard Cite

The prompt reconstruction of the data recorded from the Large Hadron Collider (LHC) detectors has always been addressed by dedicated resources at the CERN Tier-0. Such workloads come in spikes due to the nature of the operation of the accelerator and in special high load occasions experiments have commissioned methods to distribute (spill-over) a fraction of the load to sites outside CERN. The present work demonstrates a new way of supporting the Tier-0 environment by provisioning resources elastically for such spilled-over workflows onto the Piz Daint Supercomputer at CSCS. This is implemented using containers, tuning the existing batch scheduler and reinforcing the scratch file system, while still using standard Grid middleware. ATLAS, CMS and CSCS have jointly run selected prompt data reconstruction on up to several thousand cores on Piz Daint into a shared environment, thereby probing the viability of the CSCS high performance computer site as on demand extension of the CERN Tier-0, which could play a role in addressing the future LHC computing challenges for the high luminosity LHC.

show abstract

“…Almost all kinds of jobs (except for very IO intensive jobs like skims and digization using classical/non-premixed pileup) are allowed to run at NERSC. Job input that isn't available locally will be read remotely from another CMS site [3]. CHEP 2018 Figure 2.…”

Section: Hepcloud Integration Into Cms Workflow Managementmentioning

confidence: 99%

HPC resource integration into CMS Computing via HEPCloud

et al. 2019

Self Cite

View full text Add to dashboard Cite

The higher energy and luminosity from the LHC in Run 2 have put increased pressure on CMS computing resources. Extrapolating to even higher luminosities (and thus higher event complexities and trigger rates) beyond Run 3, it becomes clear that simply scaling up the the current model of CMS computing alone will become economically unfeasible. High Performance Computing (HPC) facilities, widely used in scientific computing outside of HEP, have the potential to help fill the gap. Here we describe the U.S.CMS efforts to integrate US HPC resources into CMS Computing via the HEPCloud project at Fermilab. We present advancements in our ability to use NERSC resources at scale and efforts to integrate other HPC sites as well. We present experience in the elastic use of HPC resources, quickly scaling up use when so required by CMS workflows. We also present performance studies of the CMS multi-threaded framework on both Haswell and KNL HPC resources. *

show abstract

Any Data, Any Time, Anywhere: Global Data Access for Science

Cited by 19 publications

References 12 publications

Coffea-casa: an analysis facility prototype

Coffea-casa: an analysis facility prototype

Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers

HPC resource integration into CMS Computing via HEPCloud

Contact Info

Product

Resources

About