A new era for central processing and production in CMS

Fajardo, Edgar; Gutsche, O.; Foulkes, S.; Linacre, J.; Lahiff, Andrew; Gómez-Ceballos, G.; Klute, M.; Mohapatra, A.

doi:10.1088/1742-6596/396/4/042018

Cited by 17 publications

(8 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this time it has received 3 million writes, (∼ 1 per second). Its adoption has allowed the CMS operations team to increase their efficiency [4]. Computing resource usage also increased due to the streamlined procedure for assigning workflows to agents.…”

Section: Resultsmentioning

confidence: 99%

The WorkQueue project - a task queue for the CMS workload management system

Ryu

Wakefield

2012

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

We present the development and first experience of a new component (termed WorkQueue) in the CMS workload management system. This component provides a link between a global request system (Request Manager) and agents (WMAgents) which process requests at compute and storage resources (known as sites). These requests typically consist of creation or processing of a data sample (possibly terabytes in size).Unlike the standard concept of a task queue, the WorkQueue does not contain fully resolved work units (known typically as jobs in HEP). This would require the WorkQueue to run computationally heavy algorithms that are better suited to run in the WMAgents. Instead the request specifies an algorithm that the WorkQueue uses to split the request into reasonable size chunks (known as elements). An advantage of performing lazy evaluation of an element is that expanding datasets can be accommodated by having job details resolved as late as possible.The WorkQueue architecture consists of a global WorkQueue which obtains requests from the request system, expands them and forms an element ordering based on the request priority. Each WMAgent contains a local WorkQueue which buffers work close to the agent, this overcomes temporary unavailability of the global WorkQueue and reduces latency for an agent to begin processing. Elements are pulled from the global WorkQueue to the local WorkQueue and into the WMAgent based on the estimate of the amount of work within the element and the resources available to the agent.WorkQueue is based on CouchDB, a document oriented no-sql database. WorkQueue uses the features of CouchDB (map/reduce views, bi-directional replication between distributed instances) to provide a scalable distributed system for managing large queues of work.The project described here represents an improvement over the old approach to workload management in CMS which involved individual operators feeding requests into agents. This new approach allows for a system where individual WMAgents are transient and can be added or removed from the system as needed. Presented at CHEP 2012: International Conference on Computing in High Energy and Nuclear PhysicsThe WorkQueue project -a task queue for the CMS workload management system S Ryu 1 , S Wakefield 2 1 Fermi National Laboratory, Batavia, Illinois, USA 2 Imperial College London, London, UK E-mail: stuart.wakefield@imperial.ac.ukAbstract. We present the development and first experience of a new component (termed WorkQueue) in the CMS workload management system. This component provides a link between a global request system (Request Manager) and agents (WMAgents) which process requests at compute and storage resources (known as sites). These requests typically consist of creation or processing of a data sample (possibly terabytes in size).Unlike the standard concept of a task queue, the WorkQueue does not contain fully resolved work units (known typically as jobs in HEP). This would require the WorkQueue to run computationally heavy algorithms that are better suit...

show abstract

Section: Resultsmentioning

confidence: 99%

The WorkQueue project - a task queue for the CMS workload management system

Ryu

Wakefield

2012

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

show abstract

“…The project has now turned to the use of the workow management system (WMAgent) [37] for job submission. WMAgent gives the ability to specify a destination site within the CMS infrastructure to which results are automatically replicated using the transport software PhEDEx [38].…”

Section: Compact Muon Solenoid (Cms)mentioning

confidence: 99%

LHC@Home: a BOINC-based volunteer computing infrastructure for physics studies at CERN

et al. 2017

View full text Add to dashboard Cite

Abstract:The LHC@Home BOINC project has provided computing capacity for numerical simulations to researchers at CERN since 2004, and has since 2011 been expanded with a wider range of applications. The traditional CERN accelerator physics simulation code SixTrack enjoys continuing volunteers support, and thanks to virtualisation a number of applications from the LHC experiment collaborations and particle theory groups have joined the consolidated LHC@Home BOINC project. This paper addresses the challenges related to traditional and virtualized applications in the BOINC environment, and how volunteer computing has been integrated into the overall computing strategy of the laboratory through the consolidated LHC@Home service. Thanks to the computing power provided by volunteers joining LHC@Home, numerous accelerator beam physics studies have been carried out, yielding an improved understanding of charged particle dynamics in the CERN Large Hadron Collider (LHC) and its future upgrades. The main results are highlighted in this paper.

show abstract

“…Elastic Extensions of Computing Centres on Clouds G. Codispoti management tools (WMAgent [14] for scheduled production job submissions, CRAB [15] for unpredictable user analysis job submissions). But it is not registered as a WLCG resource, as it may have a very short life.…”

Section: Pos(isgc 2016)023mentioning

confidence: 99%

Elastic Computing from Grid sites to External Clouds

Codispoti¹,

Maria²,

Aiftimiei³

et al. 2017

Proceedings of International Symposium on Grids and Clouds (ISGC) 2016 — PoS(ISGC 2016)

View full text Add to dashboard Cite

LHC experiments are now in Run-II data taking and approaching new challenges in the operation of the computing facilities in future Runs. Despite having demonstrated to be able to sustain operations at scale during Run-I, it has become evident that the computing infrastructure for Run-II already is dimensioned to cope at most with the average amount of data recorded, and not for peak usage. The latter are frequent and may create large backlogs and have a direct impact on data reconstruction completion times, hence to data availability for physics analysis. Among others, the CMS experiment is exploring (since the first Long Shutdown period after Run-I) the access and utilisation of Cloud resources provided by external partners or commercial providers. In this work we present proof of concepts of the elastic extension of a CMS Tier-3 site in Bologna (Italy), on an external OpenStack infrastructure. We start from presenting the experience on a first work on the "Cloud Bursting" of a CMS Grid site using a novel LSF configuration to dynamically register new worker nodes. Then, we move to an even more recent work on a "Cloud Site as-aService" prototype, based on a more direct access/integration of OpenStack resources into the CMS workload management system. Results with real CMS workflows and future plans are also presented and discussed.

show abstract

A new era for central processing and production in CMS

Cited by 17 publications

References 9 publications

The WorkQueue project - a task queue for the CMS workload management system

The WorkQueue project - a task queue for the CMS workload management system

LHC@Home: a BOINC-based volunteer computing infrastructure for physics studies at CERN

Elastic Computing from Grid sites to External Clouds

Contact Info

Product

Resources

About