Devising a Cloud Scientific Workflow Platform for Big Data

Abstract

Abstract-Scientificworkflow management systems (SWFMSs) are facing unprecedented challenges from big data deluge. As revising all the existing workflow applications to fit into Cloud computing paradigm is impractical, thus migrating SWFMSs into the Cloud to leverage the functionalities of both Cloud computing and SWFMSs may provide a viable approach to big data processing. In this paper, we first discuss the challenges for scientific workflow applications and the available solutions in details, and analyze the essential requirements for a scientific computing Cloud platform. Then we propose a service framework to normalize the integration of SWFMS with Cloud computing. Meanwhile, we also present our implementation experience based on the service Framework. At last, we set up a series of experiments to demonstrate the capability of our implementation and use a Montage Image Mosaic Workflow as a showcase of the implementation.

Data Placement for Privacy-Aware Applications over Big Data in Hybrid Clouds

“…= MAX , = 1 (8) while ≤ | V | do (9) if V , can hold then (10) Calculate the total cost TC by (10) (11) if TC < then (12) = TC (13) else = + 1 (14) end if (15) end if (16) end while (17) U p d a t e with cost C (18) end if (19) PM 2 separately, and the data transferring between these two VMs only employs the switch 4 .…”

“…, } (1) Algorithm 1 Special VM identification for data access (2) for = 1 to | | do (3) if needs to be placed in private cloud then (4) for = 1 to do (5) if can hold then (6) A d d to (7) end if (8) end for (9) S o r t in the decreasing order of VM distance between the VM in svi and the VM in (10) Calculate the energy consumption ec 1 after allocating to ,1 by (16) (11) = ec 1 , num = 2 (12) while num ≤ | | do (13) Calculate the energy consumption ec num after allocating to ,num by Eq. (16) (14) if ec num < then (15) = ec num (16) end if (17) n u m= num + 1 (18) end while (19) U p d a t e according to MC and the relevant PM (20) end if We use 4 datasets with different scale of datasets that need to be placed in the hybrid cloud. And 20% of them are privacyaware data, which should be placed in the private cloud.…”

“…Big data needs a huge mass of computing resources and storage resources, to promote the development of cloud technology [19][20][21]. Data placement in cloud environment has been widely concerned to improve the quality of cloud services.…”

A scientific workflow management system architecture and its scheduling based on cloud service platform for manufacturing big data analytics

“…Lin et al [8] proposed a reference architecture of SWfMSs which composes four layers, namely, presentation layer, workflow management layer, task management layer, and operational layer. Zhaoet al [6] devised a cloud scientific workflow platform for big data, and the framework includes client layer, service layer, middleware layer, and infrastructure layer. We reference the related work discussed above, and propose a prototype architecture of scientific workflow management system based on the cloud manufacturing service platform (as shown in Fig.…”

“…Besides, cloud computing which easily provides the distribute hardware and software resources as visual services for scientific researches and engineering applications can also be combined with the scientific workflows [2]. Within cloud environments, we can encapsulate resources and complex collecting, filtering, computing, visualizations, and other processing steps into different services to be orchestrated as a model of scientific workflows to be managed to process the manufacturing big data [6].…”