Automatic data placement and replication in grids

Ding, Ying; Lü, Ying

doi:10.1109/hipc.2009.5433229

Cited by 10 publications

(5 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the past decade, the data placement problem has witnessed extensive research with a wide variety of techniques developed for different execution environments, namelydistributed computing [9], [17], grid computing [13], [26], [27], and cloud computing [16], [19], [30], [44]. Initially, the focus of these works was on relational workloads such as database joins [17] and scientific workloads [14], [31], [45], however, recently the focus has shifted towards workloads emanating from specialized applications such as OSN services [21], [24] and data intensive services in geo-distributed clouds [1], [41]- [43], [47].…”

Section: Related Workmentioning

confidence: 99%

UnifyDR: A Generic Framework for Unifying Data and Replica Placement

et al. 2020

View full text Add to dashboard Cite

The advent of (big) data management applications operating at Cloud scale has led to extensive research on the data placement problem. The key objective of data placement is to obtain a partitioning (possibly allowing for replicas) of a set of data-items into distributed nodes that minimizes the overall network communication cost. Although replication is intrinsic to data placement, it has seldom been studied in combination with the latter. On the contrary, most of the existing solutions treat them as two independent problems, and employ a two-phase approach: (1) data placement, followed by (2) replica placement. We address this by proposing a new paradigm, CDR, with the objective of combining data and replica placement as a single joint optimization problem. Specifically, we study two variants of the CDR problem: (1) CDR-Single, where the objective is to minimize the communication cost alone, and (2) CDR-Multi, which performs a multi-objective optimization to also minimize traffic and storage costs. To unify data and replica placement, we propose a generic framework called UnifyDR, which leverages overlapping correlation clustering to assign a data-item to multiple nodes, thereby facilitating data and replica placement to be performed jointly. We establish the generic nature of UnifyDR by portraying its ability to address the CDR problem in two real-world use-cases, that of join-intensive online analytical processing (OLAP) queries and a location-based online social network (OSN) service. The effectiveness and scalability of UnifyDR are showcased by experiments performed on data generated using the TPC-DS benchmark and a trace of the Gowalla OSN for the OLAP queries and OSN service use-case, respectively. Empirically, the presented approach obtains an improvement of approximately 35% in terms of the evaluated metrics and a speed-up of 8 times in comparison to state-of-the-art techniques.

show abstract

Section: Related Workmentioning

confidence: 99%

UnifyDR: A Generic Framework for Unifying Data and Replica Placement

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The node at the top is treated as parent or server that is responsible for management of all the nodes beneath the root node. The advantage of this hierarchical structure is efficient usage of bandwidth, effective workload management, and scalable management of data sets and user . The problem with such topology is that there is direct communication only between the child and its immediate parent.…”

Section: Asynchronous Replication Strategiesmentioning

confidence: 99%

Strategies for replica consistency in data grid – a comprehensive survey

Vashisht¹,

Sharma

Kumar

2016

Concurrency and Computation

View full text Add to dashboard Cite

Summary Data grid provides an efficient solution for data‐oriented applications that need to manage and process large data sets located at geographically distributed storage resources. Data grid relies on data replicas to enhance the performance and to ensure the fault tolerant results to the users. Replicas are developed to increase the availability of data and to provide better data access. Replicas have their own advantages, but there are a number of issues that must be resolved. Among various existing issues, the critical concern is replica consistency. Various replica consistency strategies are available in the literature. These strategies rationalize and investigate various parameters like bandwidth consumption, access cost, scalability, execution time, storage consumption, staleness, and freshness of replicas. In this paper, several asynchronous replica consistencies are classified and analyzed based on various strategies such as topology, level of abstraction, update propagation, and locality. Some other strategies are also discussed and analyzed like adaptive consistency, quorum‐based consistency, load balancing, and agent‐based economically efficient, check‐pointing, fault tolerance, and conflict management. Parameters on which these strategies are analyzed are methodology, replication classification, consistency, grid topology, environment, evaluation parameters, and performance. Copyright © 2016 John Wiley & Sons, Ltd.

show abstract

“…Perfect Balancing (PB) [34] is a static data placement strategy for distributed storage clusters to reduce response time through load-balancing. In [35], the authors proposed a data placement algorithm which can provide grid users with automatic and intelligent data placement by considering the characteristics of the datasets such as their sizes and popularities. In [36], Cope et al proposed a series of robust data placement techniques to perform time critical workflows in heterogeneous urgent computing systems.…”

Section: Related Workmentioning

confidence: 99%

A clustering based coscheduling strategy for efficient scientific workflow execution in cloud computing

Deng

Ren

Song

et al. 2013

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARY Due to its advantages of cost‐effectiveness, on‐demand provisioning and easy for sharing, cloud computing has grown in popularity with the research community for deploying scientific applications such as workflows. Although such interests continue growing and scientific workflows are widely deployed in collaborative cloud environments that consist of a number of data centers, there is an urgent need for exploiting strategies which can place application datasets across globally distributed data centers and schedule tasks according to the data layout to reduce both latency and makespan for workflow execution. In this paper, by utilizing dependencies among datasets and tasks, we propose an efficient data and task coscheduling strategy that can place input datasets in a load balance way and meanwhile, group the mostly related datasets and tasks together. Moreover, data staging is used to overlap task execution with data transmission in order to shorten the start time of tasks. We build a simulation environment on Tianhe supercomputer for evaluating the proposed strategy and run simulations by random and realistic workflows. The results demonstrate that the proposed strategy can effectively improve scheduling performance while reducing the total volume of data transfer across data centers. Concurrency and Computation: Practice and Experience, 2013.© 2013 Wiley Periodicals, Inc.

show abstract

Automatic data placement and replication in grids

Abstract: Abstract

Cited by 10 publications

References 23 publications

UnifyDR: A Generic Framework for Unifying Data and Replica Placement

UnifyDR: A Generic Framework for Unifying Data and Replica Placement

Strategies for replica consistency in data grid – a comprehensive survey

A clustering based coscheduling strategy for efficient scientific workflow execution in cloud computing

Contact Info

Product

Resources

About