Commissioning the HTCondor-CE for the Open Science Grid

Bockelman, Brian; Cartwright, Timothy J.; Frey, Jaime; Fajardo, Edgar; Lin, Brian; Selmeci, Mátyás; Tannenbaum, Todd; Zvada, M.

doi:10.1088/1742-6596/664/6/062003

Cited by 31 publications

(24 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(), in which HTCondor (Bockelman et al. ) is used to employ multiple processors to compute the downscaling of ECMWF runoff forecast on multiple watersheds. We provide a more comprehensive parallelization to achieve both high throughput (via job scheduler and batch computing) and high performance (via TauDEM).…”

Section: Introductionmentioning

confidence: 99%

“…First, our model is designed to achieve 1/3rd arc-second or finer inundation mapping for CONUS and covers all 2.7 million reaches in NHDPlus from the beginning. Second, high-throughput computing (HTC) is the only parallel computing model used in Snow et al (2016), in which HTCondor (Bockelman et al 2015) is used to employ multiple processors to compute the downscaling of ECMWF runoff forecast on multiple watersheds. We provide a more comprehensive parallelization to achieve both high throughput (via job scheduler and batch computing) and high performance (via TauDEM).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A CyberGIS Integration and Computation Framework for High‐Resolution Continental‐Scale Flood Inundation Mapping

Liu

Maidment²,

Tarboton

et al. 2018

J American Water Resour Assoc

View full text Add to dashboard Cite

We present a Digital Elevation Model‐based hydrologic analysis methodology for continental flood inundation mapping (CFIM), implemented as a cyberGIS scientific workflow in which a 1/3rd arc‐second (10 m) height above nearest drainage (HAND) raster data for the conterminous United States (CONUS) was computed and employed for subsequent inundation mapping. A cyberGIS framework was developed to enable spatiotemporal integration and scalable computing of the entire inundation mapping process on a hybrid supercomputing architecture. The first 1/3rd arc‐second CONUS HAND raster dataset was computed in 1.5 days on the cyberGIS Resourcing Open Geospatial Education and Research supercomputer. The inundation mapping process developed in our exploratory study couples HAND with National Water Model forecast data to enable near real‐time inundation forecasts for CONUS. The computational performance of HAND and the inundation mapping process were profiled to gain insights into the computational characteristics in high‐performance parallel computing scenarios. The establishment of the CFIM computational framework has broad and significant research implications that may lead to further development and improvement of flood inundation mapping methodologies.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A CyberGIS Integration and Computation Framework for High‐Resolution Continental‐Scale Flood Inundation Mapping

Liu

Maidment²,

Tarboton

et al. 2018

J American Water Resour Assoc

View full text Add to dashboard Cite

show abstract

“…Development began on a CMS@Home project in 2015 using a modi ed CMS Remote Analysis Builder v3 (CRAB3) [32] server VM submitting jobs running CMS standard software (CMSSW) [33] to a dedicated HTCondor [34] server VM rather than the normal submission to the Worldwide LHC Computing Grid (WLCG) [35]. The VMs were run at Rutherford Appleton Laboratory (RAL), UK.…”

Section: Compact Muon Solenoid (Cms)mentioning

confidence: 99%

LHC@Home: a BOINC-based volunteer computing infrastructure for physics studies at CERN

et al. 2017

View full text Add to dashboard Cite

Abstract:The LHC@Home BOINC project has provided computing capacity for numerical simulations to researchers at CERN since 2004, and has since 2011 been expanded with a wider range of applications. The traditional CERN accelerator physics simulation code SixTrack enjoys continuing volunteers support, and thanks to virtualisation a number of applications from the LHC experiment collaborations and particle theory groups have joined the consolidated LHC@Home BOINC project. This paper addresses the challenges related to traditional and virtualized applications in the BOINC environment, and how volunteer computing has been integrated into the overall computing strategy of the laboratory through the consolidated LHC@Home service. Thanks to the computing power provided by volunteers joining LHC@Home, numerous accelerator beam physics studies have been carried out, yielding an improved understanding of charged particle dynamics in the CERN Large Hadron Collider (LHC) and its future upgrades. The main results are highlighted in this paper.

show abstract

“…Resources include clusters "owned" by CMS within the WLCG, allocations at supercomputing centers, and cloud-based resources. This provisioning is done by submitting a "glidein" to the clusters via various grid [10] and cloud interfaces (such as Amazon EC2), or even just SSH [11]. Upon starting on the resource, the glidein joins the global pool, and multiple jobs are sched-uled in parallel or in series on the resource, depending on available hardware.…”

Section: Htcondor and Glideinwmsmentioning

confidence: 99%

Any Data, Any Time, Anywhere: Global Data Access for Science

Bloom

Bockelman

Bradley

et al. 2015

2015 IEEE/ACM 2nd International Symposium on Big Data Computing (BDC)

View full text Add to dashboard Cite

Data access is key to science driven by distributed high-throughput computing (DHTC), an essential technology for many major research projects such as High Energy Physics (HEP) experiments. However, achieving efficient data access becomes quite difficult when many independent storage sites are involved because users are burdened with learning the intricacies of accessing each system and keeping careful track of data location. We present an alternate approach: the Any Data, Any Time, Anywhere infrastructure. Combining several existing software products, AAA presents a global, unified view of storage systems -a "data federation," a global filesystem for software delivery, and a workflow management system. We present how one HEP experiment, the Compact Muon Solenoid (CMS), is utilizing the AAA infrastructure and some simple performance metrics.

show abstract

Commissioning the HTCondor-CE for the Open Science Grid

Cited by 31 publications

References 9 publications

A CyberGIS Integration and Computation Framework for High‐Resolution Continental‐Scale Flood Inundation Mapping

A CyberGIS Integration and Computation Framework for High‐Resolution Continental‐Scale Flood Inundation Mapping

LHC@Home: a BOINC-based volunteer computing infrastructure for physics studies at CERN

Any Data, Any Time, Anywhere: Global Data Access for Science

Contact Info

Product

Resources

About