Experience with dynamic resource provisioning of the CMS online cluster using a cloud overlay

Abstract

The primary goal of the online cluster of the Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider (LHC) is to build event data from the detector and to select interesting collisions in the High Level Trigger (HLT) farm for offline storage. With more than 1500 nodes and a capacity of about 850 kHEPSpecInt06, the HLT machines represent similar computing capacity of all the CMS Tier1 Grid sites together. Moreover, it is currently connected to the CERN IT datacenter via a dedicated 160 Gbps network connection and hence can access the remote EOS based storage with a high bandwidth. In the last few years, a cloud overlay based on OpenStack has been commissioned to use these resources for the WLCG when they are not needed for data taking. This online cloud facility was designed for parasitic use of the HLT, which must never interfere with its primary function as part of the DAQ system. It also allows to abstract from the different types of machines and their underlying segmented networks. During the LHC technical stop periods, the HLT cloud is set to its static mode of operation where it acts like other grid facilities. The online cloud was also extended to make dynamic use of resources during periods between LHC fills. These periods are a-priori unscheduled and of undetermined length, typically of several hours, once or more a day. For that, it dynamically follows LHC beam states and hibernates Virtual Machines (VM) accordingly. Finally, this work presents the design and implementation of a mechanism to dynamically ramp up VMs when the DAQ load on the HLT reduces towards the end of the fill.

Development of the CMS detector for the CERN LHC Run 3

“…During interfill periods, week-long technical stops, and longer LHC and detector upgrade periods, this capacity is mostly unused for the HLT tasks. Thus, starting in Run 2, infrastructure was developed [276] to run an Openstack-based cloud overlay to use the computing resources for offline production jobs. In order to not overlap with the HLT-based workflows, the FUs are able to run virtual machines (VM) that provide the necessary software environment for the worldwide LHC computing grid (WLCG) CMS jobs.…”

“…The main component of the SI, the Global Pool, obtains the majority of its resources via the submission of pilot jobs to WLCG [328] and open science grid (OSG) sites. However, locally instantiated processing nodes via, for example, DODAS [373] or BOINC [374], as well as opportunistic resources, such as the HLT filter farm [276] when not in use for data taking, can also be employed. As described in section 12.7.1, the SI computing capacity has recently expanded into HPC facilities which are integrated as part of the Global or HEPCloud pools.…”

Repurposing of the Run 2 CMS High Level Trigger Infrastructure as a Cloud Resource for Offline Computing

“…The CMS Online Cloud (deployed on the HLT farm resources) was commissioned during Run 2 to dedicate HLT resources for offline data processing when not needed to support online data taking [2]. Offline computing takes place in the Global pool [3], one of the federated HT-Condor [4] pools managed by the Submission Infrastructure team in CMS Computing.…”