Cloud Bursting with GlideinWMS: Means to satisfy ever increasing computing needs for Scientific Workflows

Mhashilkar, Parag; Tiradani, Anthony; Holzman, B.; Larson, Krista; Sfiligoi, I.; Rynge, Mats

doi:10.1088/1742-6596/513/3/032069

Cited by 10 publications

(10 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We modelled a scenario in which resources from AWS, NERSC, and opportunistic resources from the OSG were available to be provisioned by glideinWMS [7] and made part of the facility's local computing cluster. We introduced jobs that expressed a preference for one of the types of resource.…”

Section: Decision Engine With Glideinwms As the Resource Provisionermentioning

confidence: 99%

HEPCloud, an Elastic Hybrid HEP Facility using an Intelligent Decision Support System

et al. 2019

Self Cite

View full text Add to dashboard Cite

HEPCloud is rapidly becoming the primary system for provisioning compute resources for all Fermilab-affiliated experiments. In order to reliably meet the peak demands of the next generation of High Energy Physics experiments, Fermilab must plan to elastically expand its computational capabilities to cover the forecasted need. Commercial cloud and allocation-based High Performance Computing (HPC) resources both have explicit and implicit costs that must be considered when deciding when to provision these resources, and at which scale. In order to support such provisioning in a manner consistent with organizational business rules and budget constraints, we have developed a modular intelligent decision support system (IDSS) to aid in the automatic provisioning of resources spanning multiple cloud providers, multiple HPC centers, and grid computing federations. In this paper, we discuss the goals and architecture of the HEPCloud Facility, the architecture of the IDSS, and our early experience in using the IDSS for automated facility expansion both at Fermi and Brookhaven National Laboratory.

show abstract

Section: Decision Engine With Glideinwms As the Resource Provisionermentioning

confidence: 99%

HEPCloud, an Elastic Hybrid HEP Facility using an Intelligent Decision Support System

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…The glideinWMS workload management system [5] was used to provision the worker nodes used during the CMS run. A development version of glideinWMS was deployed to make available some of the new features needed to run at scale.…”

Section: Scale Costs and Stability Of The Servicesmentioning

confidence: 99%

Experience in using commercial clouds in CMS

Bauerdick¹,

Bockelman²,

Dykstra³

et al. 2017

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Abstract. Historically high energy physics computing has been performed on large purposebuilt computing systems. In the beginning there were single site computing facilities, which evolved into the Worldwide LHC Computing Grid (WLCG) used today. The vast majority of the WLCG resources are used for LHC computing and the resources are scheduled to be continuously used throughout the year. In the last several years there has been an explosion in capacity and capability of commercial and academic computing clouds. Cloud resources are highly virtualized and intended to be able to be flexibly deployed for a variety of computing tasks. There is a growing interest amongst the cloud providers to demonstrate the capability to perform large scale scientific computing. In this presentation we will discuss results from the CMS experiment using the Fermilab HEPCloud Facility, which utilized both local Fermilab resources and Amazon Web Services (AWS). The goal was to work with AWS through a matching grant to demonstrate a sustained scale approximately equal to half of the worldwide processing resources available to CMS. We will discuss the planning and technical challenges involved in organizing the most IO intensive CMS workflows on a large-scale set of virtualized resource provisioned by the Fermilab HEPCloud. We will describe the data handling and data management challenges. Also, we will discuss the economic issues and cost and operational efficiency comparison to our dedicated resources. At the end we will consider the changes in the working model of HEP computing in a domain with the availability of large scale resources scheduled at peak times.

show abstract

“…Fermilab uses the GlideinWMS [19] workflow management system and the Jobsub client/server submission system to federate heterogeneous resources including grids and clouds. Users use their automatically generated SLCS-based X.509 certificates to authenticate to the server and submit jobs.…”

Section: F Scientific Workflows On Federated Cloudsmentioning

confidence: 99%

X.509 Authentication and Authorization in Fermi Cloud

Kim

Timm

2014

2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing

View full text Add to dashboard Cite

We present a summary of how X.509 authentication and authorization are used with OpenNebula in FermiCloud. We also describe a history of why the X.509 authentication was needed in FermiCloud, and review X.509 authorization options, both internal and external to OpenNebula. We show how these options can be and have been used to successfully run scientific workflows on federated clouds, which include OpenNebula on FermiCloud and Amazon Web Services as well as other community clouds. We also outline federation options being used by other commercial and open-source clouds and cloud research projects.

show abstract

Cloud Bursting with GlideinWMS: Means to satisfy ever increasing computing needs for Scientific Workflows

Cited by 10 publications

References 2 publications

HEPCloud, an Elastic Hybrid HEP Facility using an Intelligent Decision Support System

HEPCloud, an Elastic Hybrid HEP Facility using an Intelligent Decision Support System

Experience in using commercial clouds in CMS

X.509 Authentication and Authorization in Fermi Cloud

Contact Info

Product

Resources

About