Exploiting volatile opportunistic computing resources with Lobster

Woodard, A.; Wolf, M.; Mueller, C.; Tovar, Ben; Donnelly, Patrick J.; Anampa, K. Hurtado; Lannon, K.; Hildreth, M.; Thain, Douglas

doi:10.1088/1742-6596/664/3/032035

Cited by 2 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of the software components above cannot be directly installed and configured on the Notre Dame CRC campus cluster due to lack of system administrator privileges to the resource. While Lobster [11], a workflow management tool developed at the University of Notre Dame and designed to harness these opportunistic computing resources, can be used to provide the proper CMS environment locally, this does not deal with CMS grid jobs with other standard submission tools in CMS like CRAB [12].…”

Section: Deploying a Cms Tier 3 With Vc3 And The Osg Hosted Ce Servicementioning

confidence: 99%

Deploying and extending CMS Tier 3s using VC3 and the OSG Hosted CE service

et al. 2019

Self Cite

View full text Add to dashboard Cite

CMS Tier 3 centers, frequently located at universities, play an important role in the physics analysis of CMS data. Although different computing resources are often available at universities, meeting all requirements to deploy a valid Tier 3 able to run CMS workflows can be challenging in certain scenarios. For instance, providing the right operating system (OS)with access to the CERNVM File System (CVMFS) on the worker nodes or having a Compute Element (CE) on the submit host is not always allowed or possible due to e.g: lack of root access to the nodes, TCP port network policies, maintenance of a C,etc. The Notre Dame group operates a CMS Tier 3 with 1K cores. In addition to this, researchers have access to an opportunistic pool with +25K cores that are used via lobster for CMS jobs, but cannot be used with other standard CMS submission tools on the grid like CRAB, as these resources are not part of the Tier 3 due to its opportunistic nature. This work describes the use of VC3, a service for automating the deployment of virtual cluster infrastructures, in order to provide the environment (user-space CVMFS access and customized OS via singularity containers) needed for CMS workflows to work. Also, we describe its integration with the OSG Hosted CE service, to add these resources to CMS as part of our existing Tier 3 in a seamless way.

show abstract

Section: Deploying a Cms Tier 3 With Vc3 And The Osg Hosted Ce Servicementioning

confidence: 99%

Deploying and extending CMS Tier 3s using VC3 and the OSG Hosted CE service

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Tools that were available at the time like CRAB [1] or Ganga [2] did not support needed functionality like automatic re-submission and CMS dataset queries (for analysis tasks) or the ability to handle custom job description language attributes and special VOs (for Tier-1 testing purposes). Subsequently developed job management tools like CRAB3 [3] or Lobster [4] can't submit to some of the local batch systems that are available to users, enforce particular analysis environments or depend on centralized server infrastructure.…”

Section: Introductionmentioning

confidence: 99%

The swiss army knife of job submission tools: grid-control

Stober¹,

Fischer

Schleper³

et al. 2017

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

grid-control is a lightweight and highly portable open source submission tool that supports all common workflows in high energy physics (HEP). It has been used by a sizeable number of HEP analyses to process tasks that sometimes consist of up to 100k jobs.grid-control is built around a powerful plugin and configuration system, that allows users to easily specify all aspects of the desired workflow. Job submission to a wide range of local or remote batch systems or grid middleware is supported.Tasks can be conveniently specified through the parameter space that will be processed, which can consist of any number of variables and data sources with complex dependencies on each other. Dataset information is processed through a configurable pipeline of dataset filters, partition plugins and partition filters. The partition plugins can take the number of files, size of the work units, metadata or combinations thereof into account.All changes to the input datasets or variables are propagated through the processing pipeline and can transparently trigger adjustments to the parameter space and the job submission.While the core functionality is completely experiment independent, full integration with the CMS computing environment is provided by a small set of plugins.

show abstract

Exploiting volatile opportunistic computing resources with Lobster

Cited by 2 publications

References 7 publications

Deploying and extending CMS Tier 3s using VC3 and the OSG Hosted CE service

Deploying and extending CMS Tier 3s using VC3 and the OSG Hosted CE service

The swiss army knife of job submission tools: grid-control

Contact Info

Product

Resources

About