Integration of the Italian cache federation within the CMS computing model

Ciangottini, D.; Bagliesi, G.; Biasotto, M.; Cesini, Daniele; Donvito, Giacinto; Falabella, A.; Mazzoni, E.; Tracolli, Mirco

doi:10.22323/1.351.0014

Cited by 5 publications

(4 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The access to data can be performed from outer storages via remote read; still, in order to reduce the latency impact on the analysis performance, we used a local cache system. Starting from the experience achieved on data-lake prototyping at INFN, within the ESCAPE project [22], and extending the work previously done in XDC project [23], a cache solution has been integrated. The caching system must be seen as an intermediate layer between the client requesting data and the remote storage federation of the CMS experiment.…”

Section: Cachingmentioning

confidence: 99%

First experiences with a portable analysis infrastructure for LHC at INFN

et al. 2021

Self Cite

View full text Add to dashboard Cite

The challenges proposed by the HL-LHC era are not limited to the sheer amount of data to be processed: the capability of optimizing the analyser's experience will also bring important benefits for the LHC communities, in terms of total resource needs, user satisfaction and in the reduction of end time to publication. At the Italian National Institute for Nuclear Physics (INFN) a portable software stack for analysis has been proposed, based on cloud-native tools and capable of providing users with a fully integrated analysis environment for the CMS experiment. The main characterizing traits of the solution consist in the user-driven design and the portability to any cloud resource provider. All this is made possible via an evolution towards a “python-based” framework, that enables the usage of a set of open-source technologies largely adopted in both cloud-native and data-science environments. In addition, a “single sign on”-like experience is available thanks to the standards-based integration of INDIGO-IAM with all the tools. The integration of compute resources is done through the customization of a JupyterHUB solution, able to spawn identity-aware user instances ready to access data with no further setup actions. The integration with GPU resources is also available, designed to sustain more and more widespread ML based workflow. Seamless connections between the user UI and batch/big data processing framework (Spark, HTCondor) are possible. Eventually, the experiment data access latency is reduced thanks to the integrated deployment of a scalable set of caches, as developed in the context of ESCAPE project, and as such compatible with the future scenarios where a data-lake will be available for the research community. The outcome of the evaluation of such a solution in action is presented, showing how a real CMS analysis workflow can make use of the infrastructure to achieve its results.

show abstract

Section: Cachingmentioning

confidence: 99%

First experiences with a portable analysis infrastructure for LHC at INFN

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this contribution, we focus on data cache layer as meant in the above description. In Sec.2 we report on the deployment done to integrate an INFN federation of distributed caches within the "Anydata, Anytime, Anywhere (AAA)" federation [7] of CMS. This includes a summary of the studies made to measure the effect of data caches on CPU job efficiency.…”

Section: Fig 1 Estimated Cms Annual Disk Requirements Compared To Pmentioning

confidence: 99%

Smart Caching at CMS: applying AI to XCache edge services

et al. 2020

Self Cite

View full text Add to dashboard Cite

The projected Storage and Compute needs for the HL-LHC will be a factor up to 10 above what can be achieved by the evolution of current technology within a flat budget. The WLCG community is studying possible technical solutions to evolve the current computing in order to cope with the requirements; one of the main focus is resource optimization, with the ultimate aim of improving performance and efficiency, as well as simplifying and reducing operation costs. As of today the storage consolidation based on a Data Lake model is considered a good candidate for addressing HL-LHC data access challenges. The Data Lake model under evaluation can be seen as a logical system that hosts a distributed working set of analysis data. Compute power can be “close” to the lake, but also remote and thus completely external. In this context we expect data caching to play a central role as a technical solution to reduce the impact of latency and reduce network load. A geographically distributed caching layer will be functional to many satellite computing centers that might appear and disappear dynamically. In this talk we propose a system of caches, distributed at national level, describing both deployment and results of the studies made to measure the impact on the CPU efficiency. In this contribution, we also present the early results on novel caching strategy beyond the standard XRootD approach whose results will be a baseline for an AI-based smart caching system.

show abstract

“…Finally regarding data ingestion, the virtual infrastructure is configured as storage-less sites, as such input is managed through the XrootD [19] federation of CMS while the output is copied to the remote location via srm protocol, according to the job configuration. For the described pilot the actual data access happened through a XCache system deployed at INFN [20] acting as an intermediate layer for the remote data access.…”

Section: The Implemented Testbedmentioning

confidence: 99%

The DODAS Experience on the EGI Federated Cloud

et al. 2020

Self Cite

View full text Add to dashboard Cite

The EGI Cloud Compute service offers a multi-cloud IaaS federation that brings together research clouds as a scalable computing platform for research accessible with OpenID Connect Federated Identity. The federation is not limited to single sign-on, it also introduces features to facilitate the portability of applications across providers: i) a common VM image catalogue VM image replication to ensure these images will be available at providers whenever needed; ii) a GraphQL information discovery API to understand the capacities and capabilities available at each provider; and iii) integration with orchestration tools (such as Infrastructure Manager) to abstract the federation and facilitate using heterogeneous providers. EGI also monitors the correct function of every provider and collects usage information across all the infrastructure. DODAS (Dynamic On Demand Analysis Service) is an open-source Platform-as-a-Service tool, which allows to deploy software applications over heterogeneous and hybrid clouds. DODAS is one of the so-called Thematic Services of the EOSC-hub project and it instantiates on-demand container-based clusters offering a high level of abstraction to users, allowing to exploit distributed cloud infrastructures with a very limited knowledge of the underlying technologies.This work presents a comprehensive overview of DODAS integration with EGI Cloud Federation, reporting the experience of the integration with CMS Experiment submission infrastructure system.

show abstract

Integration of the Italian cache federation within the CMS computing model

Cited by 5 publications

References 3 publications

First experiences with a portable analysis infrastructure for LHC at INFN

First experiences with a portable analysis infrastructure for LHC at INFN

Smart Caching at CMS: applying AI to XCache edge services

The DODAS Experience on the EGI Federated Cloud

Contact Info

Product

Resources

About