Extending Science Gateway Frameworks to Support Big Data Applications in the Cloud

Gugnani, Shashank; Blanco, Carlos; Kiss, Tamás; Terstyánszky, Gábor

doi:10.1007/s10723-016-9369-8

Cited by 16 publications

(7 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We used a virtual machine for our BOINC clients and this system could be easily extended to use a multicloud system using the methodology described in Previti (2009) [43] if there was a need to scale the local grid system. Similarly as it can be seen in the split application, our data can follow the MapReduce paradigm and the methodology described in Gugnani et al (2016) [32] which can be used for some part of the workflow we developed. Inside the virtual machine, when a job is available, BOINC calls the application and application calls GitBox to run the shell script provided by the application.…”

Section: System Designmentioning

confidence: 99%

“…However, many of these require advanced computing skills and are difficult for nonexpert users to use. A way around this issue is to make use of science gateways [29][30][31][32][33]. These are user-friendly, easy-to-use web interfaces that enable end-user scientists to run their experiments quickly and without the need to learn the particular features of the distributed infrastructure.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Design of a Flexible, User Friendly Feature Matrix Generation System and its Application on Biomedical Datasets

et al. 2020

View full text Add to dashboard Cite

The generation of a feature matrix is the first step in conducting machine learning analyses on complex data sets such as those containing DNA, RNA or protein sequences. These matrices contain information for each object which have to be identified using complex algorithms to interrogate the data. They are normally generated by combining the results of running such algorithms across various datasets from different and distributed data sources. Thus for non-computing experts the generation of such matrices prove a barrier to employing machine learning techniques. Further since datasets are becoming larger this barrier is augmented by the limitations of the single personal computer most often used by investigators to carry out such analyses. Here we propose a user friendly system to generate feature matrices in a way that is flexible, scalable and extendable. Additionally by making use of The Berkeley Open Infrastructure for Network Computing (BOINC) software, the process can be speeded up using distributed volunteer computing possible in most institutions. The system makes use of a combination of the Grid and Cloud User Support Environment (gUSE), combined with the Web Services Parallel Grid Runtime and Developer Environment Portal (WS-PGRADE) to create workflow-based science gateways that allow users to submit work to the distributed computing. This report demonstrates the use of our proposed WS-PGRADE/gUSE BOINC system to identify features to populate matrices from very large DNA sequence data repositories, however we propose that this system could be used to analyse a wide variety of feature sets including image, numerical and text data.

show abstract

Section: System Designmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Design of a Flexible, User Friendly Feature Matrix Generation System and its Application on Biomedical Datasets

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Besides industry utilization, the usability of CCSP was also demonstrated in several academic and research application scenarios. The Raccoon2 [13] molecular docking desktop application was successfully extended with multi-cloud support as described in [14], and the automated setup and workflow-based execution of applications using Hadoop [15] was offered for users of the European Grid Infrastructure Federated Cloud (EGI FedCloud) based on the work in [16].…”

Section: Applications On the Csspmentioning

confidence: 99%

Scalable multi-cloud platform to support industry and scientific applications

Kiss

2018

2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)

Self Cite

View full text Add to dashboard Cite

Cloud computing offers resources on-demand and without large capital investments. As such, it is attractive to many industry and scientific application areas that require large computation and storage facilities. Although Infrastructure as a Service (IaaS) clouds provide elasticity and on demand resource access, the challenges represented by multi-cloud capabilities and application level scalability are still largely unsolved. The CloudSME Simulation Platform (CSSP) extended with the Microservices-based Cloud Application-level Dynamic Orchestrator (MiCADO) addresses such issues. CSSP is a generic multi-cloud access platform for the development and execution of large scale industry and scientific simulations on heterogeneous cloud resources. MiCADO provides application level scalability to optimise execution time and costs. This paper outlines how these technologies have been developed in various European research projects, and showcases several application case-studies from manufacturing, engineering and life-sciences where these tools have been successfully utilised to execute large-scale simulations in an optimised way on heterogeneous cloud infrastructures.

show abstract

“…Gugnani et al [7] investigate how workflow systems and science gateways can be extended with Big Data processing capabilities. They suggest a generic approach based on infrastructure-aware workflows and they describe an implemented proof of based on the WS-PGRADE/gUSE science gateway framework and its integration with the Hadoop parallel data processing solution based on the MapReduce paradigm in the cloud.…”

mentioning

confidence: 99%