HEPCloud, a New Paradigm for HEP Facilities: CMS Amazon Web Services Investigation

Abstract

OverviewThe use of highly distributed systems for high-throughput computing has been very successful for the broad scientific computing community. Programs such as the Open Science Grid [1] allow scientists to gain efficiency by utilizing available cycles across different domains. Traditionally, these programs have aggregated resources owned at different institutes, adding the important functionality to elastically contract and expand resources to match instantaneous demand as desired. An appealing scenario is to extend the reach of extensible resources to the rental market of commercial clouds.A prototypical example of such a scientific domain is the field of High Energy Physics (HEP), which is strongly dependent on high-throughput computing. Every stage of a modern HEP experiment requires massive resources (compute, storage, networking). Detector and simulationgenerated data have to be processed and associated with auxiliary detector and beam information to generate physics objects, which are then stored and made available to the experimenters for analysis. In the current computing paradigm, the facilities that provide the necessary resources utilize distributed high-throughput computing, with global workflow, scheduling, and data management, enabled by high-performance networks. The computing resources in these facilities are either owned by an experiment and operated by laboratories and university partners (e.g. Energy Frontier experiments at the Large Hadron Collider (LHC) such as CMS, ATLAS) or deployed for a specific program, owned and operated by the host laboratory (e.g. Intensity Frontier experiments at Fermilab such as NOvA, MicroBooNE).The HEP investment to deploy and operate these resources is significant: for example, at the time of this work, Abstract Historically, high energy physics computing has been performed on large purpose-built computing systems. These began as single-site compute facilities, but have evolved into the distributed computing grids used today. Recently, there has been an exponential increase in the capacity and capability of commercial clouds. Cloud resources are highly virtualized and intended to be able to be flexibly deployed for a variety of computing tasks. There is a growing interest among the cloud providers to demonstrate the capability to perform large-scale scientific computing. In this paper, we discuss results from the CMS experiment using the Fermilab HEPCloud facility, which utilized both local Fermilab resources and virtual machines in the Amazon Web Services Elastic Compute Cloud. We discuss the planning, technical challenges, and lessons learned involved in performing physics workflows on a large-scale set of virtualized resources. In addition, we will discuss the economics and operational efficiencies when executing workflows both in the cloud and on dedicated resources.

Optical follow-up of gravitational wave triggers with DECam during the first two LIGO/VIRGO observing runs

“…For example we have expanded the available computing resources by running the DiffImg pipeline on commercial cloud resources. As part of the Fermilab HEPCloud project (Holzman et al, 2017) we ran DiffImg processing on Amazon EC2 resources. All tests passed.…”

Cloud and on-premises data center usage, expenditures, and approaches to return on investment: A survey of academic research computing organizations

“…• The Fermi National Lab HEPCloud project did a cost analysis in 2017 that compared the economics and operational tradeoffs of cloud versus on-site HPC resources [1]. • Stewart et al analyzed ROI for three different cyberinfrastructure resources.…”

DUNE Production processing and workflow management software evaluation

“…Figure 1 illustrates the entire chain, including interaction with storage elements within the job. The architecture can also provision resources on High-Performance Computing (HPC) resources, such as Cori at the National Energy Research supercomputing Center (NERSC), within the HEP-Cloud [7,8] infrastructure. The submission mechanism is unchanged whether the jobs are HTC or HPC; this seamless transition is key to efficiently utilizing available resources and also saves the job submitter significant effort by not requiring customized submission infrastructure for different resource types.…”