The CMS Remote Analysis Builder (CRAB)

Lacaprara, S.; Bacchi, W.; Cinquilli, M.; Codispoti, G.; Corvo, M.; Dorigo, A.; Fanfani, A.; Fanzago, F.; Farina, Fabio; Merlo, Marco; Gutsche, O.; Servoli, L.; Kavka, Carlos

doi:10.1007/978-3-540-77220-0_52

Cited by 31 publications

(16 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, the data popularity service is monitoring the data access via CRAB (the CMS distributed analysis tool, see e.g. [29], [30]), identifying idle/hot data and suggesting what to clean/replicate. The two aforementioned activities are already running: despite relatively young in CMS Computing, they are growing fast and both of them will be consolidated before LHC resumes operations.…”

Section: Network Information and Analysis Throughputmentioning

confidence: 99%

Challenging data management in CMS computing with network-aware systems

Bonacorsi

Wildish

2013

2013 IEEE Nuclear Science Symposium and Medical Imaging Conference (2013 NSS/MIC)

View full text Add to dashboard Cite

After a successful first run at the LHC, and during the Long Shutdown (LS1) of the accelerator, the workload and data management sectors of the CMS Computing Model are entering into an operational review phase in order to concretely assess area of possible improvements and paths to exploit new promising technology trends. In particular, since the preparation activities for the LHC start, the Networks have constantly been of paramount importance for the execution of CMS workflows, exceeding the original expectations -as from the MONARC model -in terms of performance, stability and reliability. The low-latency transfers of PetaBytes of CMS data among dozens of WLCG Tiers worldwide using the PhEDEx dataset replication system is an example of the importance of reliable Networks. Another example is the exploitation of WAN data access over data federations in CMS. A new emerging area of work is the exploitation of ???Intelligent Network Services??, including also bandwidth on demand concepts. In this paper, we will review the work done in CMS on this, and the next steps. Abstract-After a successful first run at the LHC, and during the Long Shutdown (LS1) of the accelerator, the workload and data management sectors of the CMS Computing Model are entering into an operational review phase in order to concretely assess area of possible improvements and paths to exploit new promising technology trends. In particular, since the preparation activities for the LHC start, the networks have constantly been of paramount importance for the execution of CMS workflows, exceeding the original expectations -as from the MONARC model -in terms of performance, stability and reliability. The low-latency transfers of PetaBytes of CMS data among dozens of WLCG Tiers worldwide using the PhEDEx dataset replication system is an example of the importance of reliable networks. Another example is the exploitation of WAN data access over data federations in CMS. A new emerging area of work is the exploitation of Intelligent Network Services, including also bandwidth on demand concepts. In this paper, we will review the work done in CMS on this, and the next steps.

show abstract

Section: Network Information and Analysis Throughputmentioning

confidence: 99%

Challenging data management in CMS computing with network-aware systems

Bonacorsi

Wildish

2013

2013 IEEE Nuclear Science Symposium and Medical Imaging Conference (2013 NSS/MIC)

View full text Add to dashboard Cite

show abstract

“…The CMS Remote Analysis Builder (CRAB) [7] has been developed as a user-friendly interface to handle data analysis in a local or distributed environment, hiding the complexity of interactions with the Grid and CMS services. It allows the user to run over large distributed data samples with the same analysis code he has developed locally in a small scale test.…”

Section: Crabmentioning

confidence: 99%

“…The interaction with the Grid can be either direct with a thin CRAB client or using an intermediate CRAB Analysis Server [7] (see Fig. 2).…”

Section: Crabmentioning

confidence: 99%

Distributed Analysis in CMS

et al. 2010

View full text Add to dashboard Cite

CMS expects to manage several Pbytes of data each year, distributing them over many computing sites around the world and enabling data access at those centers for analysis. CMS has identified the distributed sites as the primary location for physics analysis to support a wide community of users, with potentially as many as 3000 users. This represents an unprecedented scale of distributed computing resources and number of users. An overview of the computing architecture, the software tools and the distributed infrastructure deployed is reported. Summaries of the experience in establishing efficient and scalable operations to prepare for CMS distributed analysis are presented, followed by the user experience in their current analysis activities.JournalofGridComputing manuscript No. (will be inserted by the editor) Abstract CMS expects to manage several Pbytes of data each year, distributing them over many computing sites around the world and enabling data access at those centers for analysis. CMS has identified the distributed sites as the primary location for physics analysis to support a wide community of users, with potentially as many as 3000 users. This represents an unprecedented scale of distributed computing resources and number of users. An overview of the computing architecture, the software tools and the distributed infrastructure deployed is reported. Summaries of the experience in establishing efficient and scalable operations to prepare for CMS distributed analysis are presented, followed by the user experience in their current analysis activities. Distributed Analysis in CMS

show abstract

“…These include generic university clusters, commercial cloud computing, and other computing infrastructure systems. Such systems are normally not prepared to run HEP workflows, as they lack the software stack, data connectivity, and infrastructure for common workflow management tools [6,7,5] to access them. Privileged access may not be available, and alterations in machine setup may not be permitted.…”

Section: Introductionmentioning

confidence: 99%

Exploiting volatile opportunistic computing resources with Lobster

Woodard

Wolf

Mueller

et al. 2015

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Abstract. Analysis of high energy physics experiments using the Compact Muon Solenoid (CMS) at the Large Hadron Collider (LHC) can be limited by availability of computing resources. As a joint effort involving computer scientists and CMS physicists at Notre Dame, we have developed an opportunistic workflow management tool, Lobster, to harvest available cycles from university campus computing pools. Lobster consists of a management server, file server, and worker processes which can be submitted to any available computing resource without requiring root access.Lobster makes use of the Work Queue system to perform task management, while the CMS specific software environment is provided via CVMFS and Parrot. Data is handled via Chirp and Hadoop for local data storage and XrootD for access to the CMS wide-area data federation. An extensive set of monitoring and diagnostic tools have been developed to facilitate system optimisation. We have tested Lobster using the 20 000-core cluster at Notre Dame, achieving approximately 8-10k tasks running simultaneously, sustaining approximately 9 Gbit/s of input data and 340 Mbit/s of output data.

show abstract

The CMS Remote Analysis Builder (CRAB)

Cited by 31 publications

References 0 publications

Challenging data management in CMS computing with network-aware systems

Challenging data management in CMS computing with network-aware systems

Distributed Analysis in CMS

Exploiting volatile opportunistic computing resources with Lobster

Contact Info

Product

Resources

About