DIRAC: a community grid solution

Tsaregorodtsev, A.; Bargiotti, M.; Brook, N. H.; Ramo, A Casajus; Castellani, G.; Charpentier, Ph.; Cioffi, Carmela; Closier, J.; Diaz, Ricardo Granciani; Kuznetsov, G; Li, Y Y; Nandakumar, R.; Paterson, S. K.; Santinelli, R.; Smith, Andrew C.; Miguelez, M S; Jimenez, S G

doi:10.1088/1742-6596/119/6/062048

Cited by 123 publications

(69 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DIRAC (Distributed Infrastructure with Remote Agent Control) [4] is a framework for distributed computing particularly well-suited to deal with large communities of users. iRODS (integrated Rule-Oriented Data System) [5] is a scalable open-source data management software used by research organizations and government agencies worldwide.…”

Section: Infraphenogrid Architecturementioning

confidence: 99%

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

Pradal

Artzet

Chopard

et al. 2017

Future Generation Computer Systems

View full text Add to dashboard Cite

Plant phenotyping consists in the observation of physical and biochemical traits of plant genotypes in response to environmental conditions. Challenges, in particular in context of climate change and food security, are numerous. High-throughput platforms have been introduced to observe the dynamic growth of a large number of plants in different environmental conditions. Instead of considering a few genotypes at a time (as it is the case when phenomic traits are measured manually), such platforms make it possible to use completely new kinds of approaches. However, the data sets produced by such widely instrumented platforms are huge, constantly augmenting and produced by increasingly complex experiments, reaching a point where distributed computation is mandatory to extract knowledge from data.In this paper, we introduce InfraPhenoGrid, the infrastructure we designed and deploy to efficiently manage data sets produced by the PhenoArch plant phenomics platform in the context of the French Phenome Project. Our solution consists in deploying scientific workflows on a Grid using a middleware to pilot workflow executions. Our approach is user-friendly in the sense that despite the intrinsic complexity of the infrastructure, running scientific workflows and understanding results obtained (using provenance information) is kept as simple as possible for end-users.

show abstract

Section: Infraphenogrid Architecturementioning

confidence: 99%

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

Pradal

Artzet

Chopard

et al. 2017

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…The possibility of having a configurable and dynamic infrastructure of computing resources is a strong motivation to explore cloud potentiality. Currently the framework adopted by JUNO to split a process in different jobs and to submit them in a distributed environment is DIRAC [12], already adopted by other experiments in the grid environment. The DIRAC framework uses the EC2 interface to instantiate virtual machines on the connected clouds.…”

Section: Juno Experimentsmentioning

confidence: 99%

The Cloud Area Padovana? An OpenStack Based IaaS for the INFN User Community

Verlato¹

2016

Proceedings of International Symposium on Grids and Clouds 2015 — PoS(ISGC2015)

View full text Add to dashboard Cite

At the end of 2013 INFN-Padova division and Legnaro National Laboratories (LNL) jointly started a new project aiming at expanding their grid-based computing and storage facility, mainly targeted to the needs of the big LHC experiments, with a cloud-based offering best suited to address the needs of the smaller sized physics experiments carried out by the local teams. Despite the great success of the grid model in supporting large scale HEP experiments, its adoption within smaller experiments has in fact been quite limited due to the well known lack of flexibility of the grid, e.g. in terms of: authentication/authorisation mechanisms; few OS and execution environments supported; batch-like only access to resources (no interactivity); deep learning curve for deploying/using services. This leads to a proliferation of several small computing clusters, disconnected from the grid infrastructure, owned by each experiment and fully dedicated to it, often underutilised but not powerful enough to satisfy peak usage needs concentrated in short periods (typically close to a scientific conference deadline). This scenario clearly implies low efficiency and large waste of both human and hardware resources for the data centre. The new cloud-based infrastructure is aimed at merging these scattered computing and storage resources in a unique facility that can serve the different experimental teams ondemand with the maximum of flexibility and elasticity made possible by the cloud paradigm. Leveraging the long-standing experience and collaboration as LHC Tier-2 of the Padova and LNL data centres, separated by 10 km but connected with a dedicated 10 Gbps optical link, the "Cloud Area Padovana" was built and put into production at the end of October 2014, after six months of pre-production operations while a couple of pilot experiments tested the capabilities of the infrastructure with real use-cases. OpenStack was chosen as Cloud Management Framework for implementing a IaaS where computing and storage resources were shared between the two data centres. However, several customisations and innovative services were added to the standard OpenStack deployment in order to address the users' needs, ensure system reliability and implement an efficient resource allocation. These concern the integration with OpenStack of authentication protocols like SAML and OpenID in order to enable user registration and access at first via INFN-AAI and later via Italian (IDEM) and possibly other international identity federations; the implementation of advanced functionalities for the management of users and projects by the IaaS administrators; the High Availability solution adopted to implement fault-tolerance of the cloud services; the development of a fair-share resource allocation mechanism analogous to the ones available in the batch system schedulers for maximizing the usage of shared resources among concurrent users/projects. An overall description of the cloud infrastructure and its operations will be given, together with the perspective of the mai...

show abstract

“…To reduce the submission overhead, pilot systems (like DIRAC [23] or Diane [24]) has been developed. In these systems, a single pilot job is submitted through gLite, as a placeholder, to a given CE.…”

Section: Related Workmentioning

confidence: 99%

Efficient extension of gLite VOs with BOINC based desktop grids

Visegrádi

Kovács

Kacsuk

2014

Future Generation Computer Systems

View full text Add to dashboard Cite

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.-The EDGI project aims to integrate Service Grid and Desktop Grid systems.-MetaJob support enables the forwarding of multiple jobs to Desktop Grids through Service Grid systems -The MetaJob provides seamless job forward since the service grid system does not recognise it as multiple job.-Seamless job forward is the key to the performance issues caused by the submission of high-number of jobs.-MetaJob can be used in any current service grids like gLite, ARC, UNICORE. Highlights (for review) Efficient extension of gLite VOs with BOINC based desktop gridsÁdám Visegrádi, József Kovács*, Peter Kacsuk MTA SZTAKI, Kende u. 13-17, 1111 Budapest, Hungary {a.visegradi, smith, kacsuk}@sztaki.hu AbstractThe paper describes the results of the EU FP7 EDGI project concerning how to extend gLite VOs with public and institutional BOINC desktop grids. Beyond simply showing the integration architecture components and services, the main emphasis is on how this integrated architecture can efficiently support parameter study applications, based on the so-called metajob concept created by the EDGI project. The paper explains in detail how to use the metajob concept by gLite users to exploit the BOINC desktop grids connected to the gLite VO, as well as how metajobs are managed internally by the 3G Bridge service. Performance measurements show that the Metajob concept indeed can significantly improve the performance of gLite VOs extended with desktop grids. Finally, the paper describes the practical ways of connecting BOINC desktop grids to gLite VOs and the accounting mechanism in these integrated grid systems.

show abstract

DIRAC: a community grid solution

Cited by 123 publications

References 15 publications

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

The Cloud Area Padovana? An OpenStack Based IaaS for the INFN User Community

Efficient extension of gLite VOs with BOINC based desktop grids

Contact Info

Product

Resources

About