The Evolution of Cloud Computing in ATLAS

Taylor, R.; Berghaus, F.; Brasolin, Franco; Cordeiro, Cristovao; Desmarais, Ron; Field, Laurence; Gable, Ian; Giordano, D.; Girolamo, A. Di; Hover, John; LeBlanc, M.; Love, P. A.; Paterson, Michael; Sobie, R.; Zaytsev, A

doi:10.1088/1742-6596/664/2/022038

Cited by 13 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cloudscheduler has already been operating an opportunistic system of distributed cloud resources in the ATLAS and Belle-II distributed computing systems. It has a proven track record of managing cloud resources for batch workload execution [11,12]. As illustrated in figure 2 Cloudscheduler needs to communicate with both the cloud interfaces and the batch system.…”

Section: Cloudschedulermentioning

confidence: 99%

Sim@P1: Using Cloudscheduler for offline processing on the ATLAS HLT farm

et al. 2019

Self Cite

View full text Add to dashboard Cite

The Simulation at Point1 (Sim@P1) project was established in 2013 to take advantage of the Trigger and Data Acquisition High Level Trigger (HLT) farm of the ATLAS experiment at the LHC. The HLT farm is a significant compute resource, which is critical to ATLAS during data taking. This large compute resource is used to generate and process simulation data for the experiment when ATLAS is not recording data. The Sim@P1 system uses virtual machines, deployed by OpenStack, in order to isolate the resources from the ATLAS technical and control network. During the upcoming long shutdown in 2019 (LS2), the HLT farm including the Sim@P1 infrastructure will be upgraded. A previous paper on the project emphasized the need for "simple, reliable, and efficient tools" to quickly switch between data acquisition operation and offline processing. In this contribution we assess various options for updating and simplifying the provisional tools. Cloudscheduler is a tool for provisioning cloud resources for batch computing that has been managing cloud resources in HEP offline computing since 2012. We present the argument for choosing Cloudscheduler, and describe technical details regarding optimal utilization of the Sim@P1 resources. *

show abstract

Section: Cloudschedulermentioning

confidence: 99%

Sim@P1: Using Cloudscheduler for offline processing on the ATLAS HLT farm

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…The HEP field has made significant inroads in utilizing elastic resources. Earlier work includes [9,10,11,12]; contrasting with the HEPCloud approach, these papers survey the status of cloud use and cost comparisons from an experiment point-of-view.…”

Section: Overviewmentioning

confidence: 99%

HEPCloud, a New Paradigm for HEP Facilities: CMS Amazon Web Services Investigation

Holzman¹,

Girone²,

Hufnagel³

et al. 2017

Comput Softw Big Sci

View full text Add to dashboard Cite

OverviewThe use of highly distributed systems for high-throughput computing has been very successful for the broad scientific computing community. Programs such as the Open Science Grid [1] allow scientists to gain efficiency by utilizing available cycles across different domains. Traditionally, these programs have aggregated resources owned at different institutes, adding the important functionality to elastically contract and expand resources to match instantaneous demand as desired. An appealing scenario is to extend the reach of extensible resources to the rental market of commercial clouds.A prototypical example of such a scientific domain is the field of High Energy Physics (HEP), which is strongly dependent on high-throughput computing. Every stage of a modern HEP experiment requires massive resources (compute, storage, networking). Detector and simulationgenerated data have to be processed and associated with auxiliary detector and beam information to generate physics objects, which are then stored and made available to the experimenters for analysis. In the current computing paradigm, the facilities that provide the necessary resources utilize distributed high-throughput computing, with global workflow, scheduling, and data management, enabled by high-performance networks. The computing resources in these facilities are either owned by an experiment and operated by laboratories and university partners (e.g. Energy Frontier experiments at the Large Hadron Collider (LHC) such as CMS, ATLAS) or deployed for a specific program, owned and operated by the host laboratory (e.g. Intensity Frontier experiments at Fermilab such as NOvA, MicroBooNE).The HEP investment to deploy and operate these resources is significant: for example, at the time of this work, Abstract Historically, high energy physics computing has been performed on large purpose-built computing systems. These began as single-site compute facilities, but have evolved into the distributed computing grids used today. Recently, there has been an exponential increase in the capacity and capability of commercial clouds. Cloud resources are highly virtualized and intended to be able to be flexibly deployed for a variety of computing tasks. There is a growing interest among the cloud providers to demonstrate the capability to perform large-scale scientific computing. In this paper, we discuss results from the CMS experiment using the Fermilab HEPCloud facility, which utilized both local Fermilab resources and virtual machines in the Amazon Web Services Elastic Compute Cloud. We discuss the planning, technical challenges, and lessons learned involved in performing physics workflows on a large-scale set of virtualized resources. In addition, we will discuss the economics and operational efficiencies when executing workflows both in the cloud and on dedicated resources.

show abstract

“…Many scientific communities have embraced or investigated the use of cloud resources for their research, including particle physics (Sadooghi et al 2015), astrophysics (Smith 2011), high-energy physics (Segal et al 2010; Taylor et al 2015), computational chemistry (Thackston and Fortenberry 2015b), chemical modeling for high–throughput drug discovery (Moghadam et al 2015), bioinformatics (Hanson et al 2014), medical imaging (Kagadis et al 2013), geophysics (Mudge et al 2011), social sciences (Wittek and Rubio-Campillo 2012), geochemistry (Huang et al 2014), genomic analysis (Ban et al 2015), and various projects at the Department of Energy (Yelick et al 2011). …”

Section: The Cloud In Scientific Researchmentioning

confidence: 99%

Automating NEURON Simulation Deployment in Cloud Resources

Stockton

Santamarı́a

2016

Neuroinform

View full text Add to dashboard Cite

Simulations in neuroscience are performed on local servers or High Performance Computing (HPC) facilities. Recently, cloud computing has emerged as a potential computational platform for neuroscience simulation. In this paper we compare and contrast HPC and cloud resources for scientific computation, then report how we deployed NEURON, a widely used simulator of neuronal activity, in three clouds: Chameleon Cloud, a hybrid private academic cloud for cloud technology research based on the Open-Stack software; Rackspace, a public commercial cloud, also based on OpenStack; and Amazon Elastic Cloud Computing, based on Amazon’s proprietary software. We describe the manual procedures and how to automate cloud operations. We describe extending our simulation automation software called NeuroManager (Stockton and Santamaria, Frontiers in Neuroinformatics, 2015), so that the user is capable of recruiting private cloud, public cloud, HPC, and local servers simultaneously with a simple common interface. We conclude by performing several studies in which we examine speedup, efficiency, total session time, and cost for sets of simulations of a published NEURON model.

show abstract

The Evolution of Cloud Computing in ATLAS

Cited by 13 publications

References 16 publications

Sim@P1: Using Cloudscheduler for offline processing on the ATLAS HLT farm

Sim@P1: Using Cloudscheduler for offline processing on the ATLAS HLT farm

HEPCloud, a New Paradigm for HEP Facilities: CMS Amazon Web Services Investigation

Automating NEURON Simulation Deployment in Cloud Resources

Contact Info

Product

Resources

About