OASIS: a data and software distribution service for Open Science Grid

Bockelman, Brian; Bejar, J. Caballero; Stefano, J De; Hover, John; Quick, Robert; Teige, S.

doi:10.1088/1742-6596/513/3/032013

Cited by 4 publications

(4 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, the LIGO user sees the aggregated resources as a single coherent "site, " despite it being virtualized across the grid. We utilize a central CVMFS repository [3] to distribute the LIGO so ware, providing users with uniform access to necessary so ware and small amounts of auxiliary data. e original setup of the data distribution for LIGO was purposely simple: just as the OSG pool appears to the user as a single large computational resource, we would run a central storage service and use it as if it were a large shared lesystem connected to a large cluster.…”

Section: Centralized Osg Implementationmentioning

confidence: 99%

See 1 more Smart Citation

Data Access for LIGO on the OSG

Weitzel

Bockelman

Brown

et al. 2017

Practice and Experience in Advanced Research Computing 2017: Sustainability, Success and Impact

View full text Add to dashboard Cite

During 2015 and 2016, the Laser Interferometer Gravitational-Wave Observatory (LIGO) conducted a three-month observing campaign. ese observations delivered the rst direct detection of gravitational waves from binary black hole mergers. To search for these signals, the LIGO Scienti c Collaboration uses the PyCBC search pipeline. To deliver science results in a timely manner, LIGO collaborated with the Open Science Grid (OSG) to distribute the required computation across a series of dedicated, opportunistic, and allocated resources. To deliver the petabytes necessary for such a largescale computation, our team deployed a distributed data access infrastructure based on the XRootD server suite and the CernVM File System (CVMFS). is data access strategy grew from simply accessing remote storage to a POSIX-based interface underpinned by distributed, secure caches across the OSG.Historically, the OSG has focused on providing a high-level computational services such as the dynamic creation of batch systems across multiple sites utilizing the pilot model [17]. In the pilot model, a central factory performs remote submission of batch jobs

show abstract

Section: Centralized Osg Implementationmentioning

confidence: 99%

Section: Centralized Osg Implementationmentioning

confidence: 99%

Data Access for LIGO on the OSG

Weitzel

Bockelman

Brown

et al. 2017

Practice and Experience in Advanced Research Computing 2017: Sustainability, Success and Impact

View full text Add to dashboard Cite

show abstract

“…For the LHC experiments with their world-wide distributed grid computing infrastructure, we managed to reduce the time between building a new software release and the rollout from several days to less than one hour. As a result there is now a big increase in the number of scientific users in high-energy physics and in various other fields [7], [8].…”

Section: A Time Machine For the Data Analysis Environmentmentioning

confidence: 99%

The Need for a Versioned Data Analysis Software Environment

Blomer,

Berzano,

Buncic

et al. 2014

Preprint

View full text Add to dashboard Cite

Scientific results in high-energy physics and in many other fields often rely on complex software stacks. In order to support reproducibility and scrutiny of the results, it is good practice to use open source software and to cite software packages and versions. With evergrowing complexity of scientific software on one side and with IT life-cycles of only a few years on the other side, however, it turns out that despite source code availability the setup and the validation of a minimal usable analysis environment can easily become prohibitively expensive. We argue that there is a substantial gap between merely having access to versioned source code and the ability to create a data analysis runtime environment. In order to preserve all the different variants of the data analysis runtime environment, we developed a snapshotting file system optimized for software distribution. We report on our experience in preserving the analysis environment for high-energy physics such as the software landscape used to discover the Higgs boson at the Large Hadron Collider.

show abstract

“…But, the HTTP proxies have well known limitations [15]. One such limitation is that the proxies are optimized for small files such as software [4] and experiment conditions [11] rather than the multigigabyte files that some users require. In this paper, we will present performance metrics for the use of StashCache over HTTP proxies for large files.…”

Section: Introductionmentioning

confidence: 99%

StashCache

Weitzel

Zvada

Vukotić

et al. 2019

Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (Learning)

View full text Add to dashboard Cite

Data distribution for opportunistic users is challenging as they neither own the computing resources they are using or any nearby storage. Users are motivated to use opportunistic computing to expand their data processing capacity, but they require storage and fast networking to distribute data to that processing. Since it requires significant management overhead, it is rare for resource providers to allow opportunistic access to storage. Additionally, in order to use opportunistic storage at several distributed sites, users assume the responsibility to maintain their data.In this paper we present StashCache, a distributed caching federation that enables opportunistic users to utilize nearby opportunistic storage. StashCache is comprised of four components: data origins, redirectors, caches, and clients. StashCache has been deployed in the Open Science Grid for several years and has been used by many projects. Caches are deployed in geographically distributed locations across the U.S. and Europe. We will present the architecture of StashCache, as well as utilization information of the infrastructure. We will also present performance analysis comparing distributed HTTP Proxies vs StashCache.

show abstract

OASIS: a data and software distribution service for Open Science Grid

Cited by 4 publications

References 2 publications

Data Access for LIGO on the OSG

Data Access for LIGO on the OSG

The Need for a Versioned Data Analysis Software Environment

StashCache

Contact Info

Product

Resources

About