Many-task computing for grids and supercomputers

Raicu, Ioan; Foster, Ian; Zhao, Yong

doi:10.1109/mtags.2008.4777912

Cited by 214 publications

(173 citation statements)

References 33 publications

Supporting

Mentioning

167

Contrasting

Order By: Relevance

“…[1] can be decomposed to a set of small jobs with dependencies, whose running time is short; while batch jobs in HTC are independent and running times of jobs are varying. 3) Evaluation metrics: A HTC service provider concerns the throughput in terms of the number of completed jobs over a long period of time; while a MTC service provider concerns the throughput over a short period of time.…”

Section: The Main Components Of Csfmentioning

confidence: 99%

“…For the MTC workload, firstly we generate the job flow according to dependency constraints, and then we choose the FCFS (First Come First Served) scheduling policy in DawningCloud, Evangelinos's system and the dedicated cluster system, respectively. According to [1], we set the scheduling cycle of the HTC and MTC schedulers as 60 seconds and 1 second, respectively. Deelman's system uses no scheduling policy, since all jobs run immediately without queuing.…”

Section: Emulation Experiments Configurationsmentioning

confidence: 99%

“…Third, state-of-the-art and state-of-the-practice public cloud solutions [3] [5] provide limited supports for scientific communities as service providers in terms of system controls 1 . For example, Deelman et al [5] propose that each staff of an organization (as end users) directly leases virtual machine resources from a public cloud provider--EC2 in a specified period for running applications.…”

Section: Introductionmentioning

confidence: 99%

“…We take a bottom-up approach to building DawningCloud, and present a layered architecture: the lower one is the common management service framework (in short, CSF) for the resource provider, and the upper one is a lightweight service management layer that is responsible for managing resources and workloads, which we call thin runtime environment software (in short, TRE) in this paper. CSF facilitates building TRE for heterogeneous workloads, and we have built two TRE for two typical scientific workloads: high throughput computing (HTC), and many task computing (MTC) [1]. We are also integrating other data-parallel programming models built in our previous work [35], including MapReduce [48], Dryadlike data flow [49], and All-Pairs [50].…”

Section: Introductionmentioning

confidence: 99%

“…When a resource provider adopts DawningCloud, the CSF is predeployed and running on a cloud site before 1. We give some examples of system controls.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

In Cloud, Can Scientific Communities Benefit from the Economies of Scale?

Wang

Zhan

Shi

et al. 2012

IEEE Trans. Parallel Distrib. Syst.

101

View full text Add to dashboard Cite

Abstract-The basic idea behind cloud computing is that resource providers offer elastic resources to end users. In this paper, we intend to answer one key question to the success of cloud computing: in cloud, can small-to-medium scale scientific communities benefit from the economies of scale? Our research contributions are three-fold: first, we propose an innovative public cloud usage model for small-to-medium scale scientific communities to utilize elastic resources on a public cloud site while maintaining their flexible system controls, i.e., create, activate, suspend, resume, deactivate, and destroy their high-level management entities-service management layers without knowing the details of management. Second, we design and implement an innovative system-DawningCloud, at the core of which are lightweight service management layers running on top of a common management service framework. The common management service framework of DawningCloud not only facilitates building lightweight service management layers for heterogeneous workloads, but also makes their management tasks simple. Third, we evaluate the systems comprehensively using both emulation and real experiments. We found that for four traces of two typical scientific workloads: high throughput computing (HTC) and many task computing (MTC), DawningCloud saves the resource consumption maximally by 59.5% and 72.6% for HTC and MTC service providers, respectively, and saves the total resource consumption maximally by 54% for the resource provider with respect to the previous two public cloud solutions. To this end, we conclude that small-to-medium scale scientific communities indeed can benefit from the economies of scale of public clouds with the support of the enabling system.

show abstract

Section: The Main Components Of Csfmentioning

confidence: 99%

Section: Emulation Experiments Configurationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…When a resource provider adopts DawningCloud, the CSF is predeployed and running on a cloud site before 1. We give some examples of system controls.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

In Cloud, Can Scientific Communities Benefit from the Economies of Scale?

Wang

Zhan

Shi

et al. 2012

IEEE Trans. Parallel Distrib. Syst.

101

View full text Add to dashboard Cite

show abstract

Many task computing for orthologous genes identification in protozoan genomes using Hydra

Coutinho

Ogasawara

Oliveira

et al. 2011

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARYOne of the main advantages of using a scientific workflow management system (SWfMS) is to orchestrate data flows among scientific activities and register provenance of the whole workflow execution. Nevertheless, the execution control of distributed activities in high performance computing environments by SWfMS presents challenges such as steering control and provenance gathering. Such challenges may become a complex task to be accomplished in bioinformatics experiments, particularly in Many Task Computing scenarios. This paper presents a data parallelism solution for a bioinformatics experiment supported by Hydra, a middleware that bridges SWfMS and high performance computing to enable workflow parallelization with provenance gathering. Hydra Many Task Computing parallelization strategies can be registered and reused. Using Hydra, provenance may also be uniformly gathered. We have evaluated Hydra using an Orthologous Gene Identification workflow. Experimental results show that a systematic approach for distributing parallel activities is viable, sparing scientist time and diminishing operational errors, with the additional benefits of distributed provenance support.

show abstract

An adaptive parallel execution strategy for cloud‐based scientific workflows

Oliveira

Ogasawara

Ocaña

et al. 2011

Concurrency and Computation

View full text Add to dashboard Cite

Many of the existing large-scale scientific experiments modeled as scientific workflows are computeintensive. Some scientific workflow management systems already explore parallel techniques, such as parameter sweep and data fragmentation, to improve performance. In those systems, computing resources are used to accomplish many computational tasks in high performance environments, such as multiprocessor machines or clusters. Meanwhile, cloud computing provides scalable and elastic resources that can be instantiated on demand during the course of a scientific experiment, without requiring its users to acquire expensive infrastructure or to configure many pieces of software. In fact, because of these advantages some scientists have already adopted the cloud model in their scientific experiments. However, this model also raises many challenges. When scientists are executing scientific workflows that require parallelism, it is hard to decide a priori the amount of resources to use and how long they will be needed because the allocation of these resources is elastic and based on demand. In addition, scientists have to manage new aspects such as initialization of virtual machines and impact of data staging. SciCumulus is a middleware that manages the parallel execution of scientific workflows in cloud environments. In this paper, we introduce an adaptive approach for executing parallel scientific workflows in the cloud. This approach adapts itself according to the availability of resources during workflow execution. It checks the available computational power and dynamically tunes the workflow activity size to achieve better performance. Experimental evaluation showed the benefits of parallelizing scientific workflows using the adaptive approach of SciCumulus, which presented an increase of performance up to 47.1%.

show abstract

Many-task computing for grids and supercomputers

Abstract: Abstract

Cited by 214 publications

References 33 publications

In Cloud, Can Scientific Communities Benefit from the Economies of Scale?

In Cloud, Can Scientific Communities Benefit from the Economies of Scale?

Many task computing for orthologous genes identification in protozoan genomes using Hydra

An adaptive parallel execution strategy for cloud‐based scientific workflows

Contact Info

Product

Resources

About