CernVM Co-Pilot: an Extensible Framework for Building Scalable Computing Infrastructures on the Cloud

Harutyunyan, A.; Blomer, Jakob; Bunčić, P.; Charalampidis, Ioannis; Grey, François; Karneyeu, A.; Larsen, D.; González, Daniel Lombraña; Lisec, J; Segal, Ben

doi:10.1088/1742-6596/396/3/032054

Cited by 11 publications

(7 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…lhc@home 2.0 in turn relies on cernvm [34,35] (a Virtual-Machine computing environment based on Scientific Linux), on the copilot job submission system [36,35], and on the Boinc middleware for volunteer computing [37,38].…”

Section: Introductionmentioning

confidence: 99%

MCPLOTS: a particle physics resource based on volunteer computing

et al. 2014

View full text Add to dashboard Cite

The mcplots.cern.ch web site (mcplots) provides a simple online repository of plots made with highenergy-physics event generators, comparing them to a wide variety of experimental data. The repository is based on the hepdata online database of experimental results and on the rivet Monte Carlo analysis tool. The repository is continually updated and relies on computing power donated by volunteers, via the lhc@home 2.0 platform.

show abstract

Section: Introductionmentioning

confidence: 99%

MCPLOTS: a particle physics resource based on volunteer computing

et al. 2014

View full text Add to dashboard Cite

show abstract

“…Most of these systems are tailored to specific use cases, workloads, resources, interfaces or development models. Some notable examples are: (i) HTCondor with Glidein on OSG [22], one of the most widely used pilot systems for the execution of mostly single-core tasks; (ii) the pilot systems developed for the LHC communities (e.g., PanDA [23], GlideinWMS [24], DIRAC [25] and CernVM Co-Pilot [26]) which execute millions of jobs a week and are specialized in supporting Large Hadron Collider (LHC) workloads on specific resources like those of the Worldwide LHC Computing Grid and the CERN cloud infrastructure; (iii) Falkon [27], specifically designed to support function-level parallelism as opposed to process-level parallelism; (iv) FireWorks [28], designed to support function-level parallelism and small-scale process-level parallelism on HPC resources; and (v) GWpilot [29] that enables the use of arbitrary scheduling algorithms with the GridWay meta-scheduler, and supports a limited number of non-MPI use cases.…”

Section: Related Workmentioning

confidence: 99%

Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms

Merzky¹,

Turilli²,

Титов³

et al. 2021

Preprint

View full text Add to dashboard Cite

Many extreme scale scientific applications have workloads comprised of a large number of individual highperformance tasks. The Pilot abstraction decouples workload specification, resource management, and task execution via job placeholders and late-binding. As such, suitable implementations of the Pilot abstraction can support the collective execution of large number of tasks on supercomputers. We introduce RADICAL-Pilot (RP) as a portable, modular and extensible Pilot enabled runtime system. We describe RP's design, architecture and implementation. We characterize its performance and show its ability to scalably execute workloads comprised of tens of thousands heterogeneous tasks on DOE and NSF leadership-class HPC platforms. Specifically, we investigate RP's weak/strong scaling with CPU/GPU, single/multi core, (non)MPI tasks and python functions when using most of ORNL Summit and TACC Frontera. RADICAL-Pilot can be used stand-alone, as well as the runtime for third-party workflow systems.

show abstract

“…The LHC Pilot-Job systems have been designed to be functionally very similar, work on almost the same underlying infrastructure, and serve applications with very similar characteristics. Around 2011, these similarities enabled Co-Pilot [90,91] to support the execution of resource placeholders on cloud and volunteer computing [92] resource pools for all the LHC experiments.…”

Section: Chronological Evolutionmentioning

confidence: 99%

A Comprehensive Perspective on Pilot-Job Systems

Turilli

Santcroos²,

Jha

2018

ACM Comput. Surv.

View full text Add to dashboard Cite

Pilot-Job systems play an important role in supporting distributed scientific computing. They are used to consume more than 700 million CPU hours a year by the Open Science Grid communities, and by processing up to 1 million jobs a day for the ATLAS experiment on the Worldwide LHC Computing Grid. With the increasing importance of task-level parallelism in high-performance computing, Pilot-Job systems are also witnessing an adoption beyond traditional domains. Notwithstanding the growing impact on scientific research, there is no agreement upon a definition of Pilot-Job system and no clear understanding of the underlying abstraction and paradigm. Pilot-Job implementations have proliferated with no shared best practices or open interfaces and little interoperability. Ultimately, this is hindering the realization of the full impact of Pilot-Jobs by limiting their robustness, portability, and maintainability. This paper offers a comprehensive analysis of Pilot-Job systems critically assessing their motivations, evolution, properties, and implementation. The three main contributions of this paper are: (i) an analysis of the motivations and evolution of Pilot-Job systems; (ii) an outline of the Pilot abstraction, its distinguishing logical components and functionalities, its terminology, and its architecture pattern; and (iii) the description of core and auxiliary properties of Pilot-Jobs systems and the analysis of seven exemplar Pilot-Job implementations. Together, these contributions illustrate the Pilot paradigm, its generality, and how it helps to address some challenges in distributed scientific computing.

show abstract

CernVM Co-Pilot: an Extensible Framework for Building Scalable Computing Infrastructures on the Cloud

Cited by 11 publications

References 3 publications

MCPLOTS: a particle physics resource based on volunteer computing

MCPLOTS: a particle physics resource based on volunteer computing

Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms

A Comprehensive Perspective on Pilot-Job Systems

Contact Info

Product

Resources

About