A fully unprivileged CernVM-FS

Blomer, Jakob; Dykstra, Dave; Ganis, G.; Mosciatti, Simone; Priessnitz, Jan

doi:10.1051/epjconf/202024507012

Cited by 5 publications

(4 citation statements)

References 6 publications

(4 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The program requires a specific environment to work correctly: (i) external connectivity; (ii) the fusermount library or unprivileged namespace mount points or a setuid installation of Singularity/Apptainer. Blomer et al provide further details on the package [13].…”

Section: Software Dependencies Through Cvmfsmentioning

confidence: 99%

Integrating LHCb Offline Workflows on Supercomputers State of Practice

Boyer,

Stagni,

Haen

et al. 2024

EPJ Web of Conf.

View full text Add to dashboard Cite

To better understand experimental conditions and performances of its experiment, the LHCb collaboration executes tens of thousands of looselycoupled and CPU-intensive Monte Carlo simulation workflows per hour. To meet the increasing LHC computing needs, funding agencies encourage the collaboration to exploit High-Performance Computing resources, and more specifically supercomputers, which offer a significant additional amount of computing resources but also come with higher integration challenges. This state-ofpractice paper outlines years of integration of LHCb simulation workflows on several supercomputers. The main contributions of this paper are: (i) an extensive description of the gap to address to run High-Energy Physics Monte Carlo simulation workflows on supercomputers; (ii) various methods and proposals to maximize the use of allocated CPU resources; (iii) a comprehensive analysis of LHCb production workflows running on diverse supercomputers.

show abstract

Section: Software Dependencies Through Cvmfsmentioning

confidence: 99%

Integrating LHCb Offline Workflows on Supercomputers State of Practice

Boyer,

Stagni,

Haen

et al. 2024

EPJ Web of Conf.

View full text Add to dashboard Cite

show abstract

“…Second is the configuration of the HPC runtime environment through the use of containers (they all support Singularity, except NERSC, where shifter is used). In addition, CVMFS [4] is available at all these HPC sites, either provided directly by the site or by mounting it in user space with the cvmfsexec package [5]. All these HPC sites provide outbound internet connectivity from the worker nodes as well, which makes integration a lot easier.…”

Section: Hepcloudmentioning

confidence: 99%

Extending the distributed computing infrastructure of the CMS experiment with HPC resources

Adelman-McCarthy

Caspart²,

Peris³

et al. 2023

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Particle accelerators are an important tool to study the fundamental properties of elementary particles. Currently the highest energy accelerator is the LHC at CERN, in Geneva, Switzerland. Each of its four major detectors, such as the CMS detector, produces dozens of Petabytes of data per year to be analyzed by a large international collaboration. The processing is carried out on the Worldwide LHC Computing Grid, that spans over more than 170 compute centers around the world and is used by a number of particle physics experiments. Recently the LHC experiments were encouraged to make increasing use of HPC resources. While Grid resources are homogeneous with respect to the used Grid middleware, HPC installations can be very different in their setup. In order to integrate HPC resources into the highly automatized processing setups of the CMS experiment a number of challenges need to be addressed. For processing, access to primary data and metadata as well as access to the software is required. At Grid sites all this is achieved via a number of services that are provided by each center. However at HPC sites many of these capabilities cannot be easily provided and have to be enabled in the user space or enabled by other means. At HPC centers there are often restrictions regarding network access to remote services, which is again a severe limitation. The paper discusses a number of solutions and recent experiences by the CMS experiment to include HPC resources in processing campaigns.

show abstract

“…The program needs a specific environment to work correctly: (i) external connectivity; (ii) the fusermount library or unprivileged namespace mount points or a setuid installation of Singularity (efficient High-Performance Computing container technology). Blomer et al provide additional details about the package [10].…”

Section: Software Delivery On Supercomputersmentioning

confidence: 99%

A Subset of the CERN Virtual Machine File System: Fast Delivering of Complex Software Stacks for Supercomputing Resources

Boyer

Haen

Stagni

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Delivering a reproducible environment along with complex and up-to-date software stacks on thousands of distributed and heterogeneous worker nodes is a critical task. The CernVM-File System (CVMFS) has been designed to help various communities to deploy software on worldwide distributed computing infrastructures by decoupling the software from the Operating System. However, the installation of this file system depends on a collaboration with system administrators of the remote resources and an HTTP connectivity to fetch dependencies from external sources. Supercomputers, which offer tremendous computing power, generally have more restrictive policies than grid sites and do not easily provide the mandatory conditions to exploit CVMFS. Different solutions have been developed to tackle the issue, but they are often specific to a scientific community and do not deal with the problem in its globality. In this paper, we provide a generic utility to assist any community in the installation of complex software dependencies on supercomputers with no external connectivity. The approach consists in capturing dependencies of applications of interests, building a subset of dependencies, testing it in a given environment, and deploying it to a remote computing resource. We experiment this proposal with a real use case by exporting Gauss -a Monte-Carlo simulation program from the LHCb experiment -on Mare Nostrum, one of the top supercomputers of the world. We provide steps to encapsulate the minimum required files and deliver a light and easy-to-update subset of CVMFS: 12.4 Gigabytes instead of 5.2 Terabytes for the whole LHCb repository.

show abstract

A fully unprivileged CernVM-FS

Cited by 5 publications

References 6 publications

Integrating LHCb Offline Workflows on Supercomputers State of Practice

Integrating LHCb Offline Workflows on Supercomputers State of Practice

Extending the distributed computing infrastructure of the CMS experiment with HPC resources

A Subset of the CERN Virtual Machine File System: Fast Delivering of Complex Software Stacks for Supercomputing Resources

Contact Info

Product

Resources

About