Community Training: Partitioning Schemes in Good Shape for Federated Data Grids

Abstract

In federated Data Grids, individual institutions share their data sets within a community to enable collaborative data analysis. Data access needs to be provided in a scalable fashion since in most e-science communities, data sets do not only grow exponentially but also experience an increasing popularity. If data autonomy is retained, each individual institution has to ensure efficient access to its data. Analyzing application-specific data properties (such as data skew) or query characteristics (query patterns) and distributing data within Data Grids accordingly, allows for improved throughput for data-intensive applications and enables better load-balancing between shared resources. We propose a framework for investigating application-specific index structures for creating suitable partitioning schemes. We evaluate two variants of the well-known Quadtree data structure as well as the Zones approach, an index structure from the astrophysics domain, according to several criteria. Our framework improves data access within federated Data Grids and can be combined with well-established Grid methods as well as with more flexible P2P technologies.

Workload-aware data partitioning in community-driven data grids

“…In Section 4, we evaluate our approach using quadtree-based partitioning schemes introduced in previous work [22]. We use a sample of one million queries from an SDSS query trace on three observational data sets and a synthetic workload on a uniform data sample from the Millennium 1 simulation.…”

“…They split densely populated areas more often, resulting in approximately even data distribution across all leaves. In previous work [22], we describe the evaluation of quadtree-based partitioning schemes using our training framework in more detail.…”

“…We constructed quadtree-based partitioning schemes using the standard splitting strategy as well as the median-based heuristics from [22]. Here, we only discuss results with standard quadtrees.…”

“…Community-driven data grids have a main focus on this multitude of data repositories, where institutions are no longer interested in keeping them "private" and do not enforce data autonomy but cooperatively share data with colleagues. Previous work [22] has described a framework for comparing different partitioning techniques and discussed several measures for evaluating the partitioning quality with regard to data load balancing. It also discusses two quadtree-based partitioning schemes, without and with a median-based heuristic, which have been extended to support our weight functions and have been used in our evaluation experiments.…”

Community-driven data grids

“…The regular shape of the data partitions (rectangles in the two-dimensional case) additionally supports the efficient calculation of relevant data regions during query processing. In previous work [20], we conducted an extensive evaluation of the training phase, comparing various data structures and defining several statistics to support the communities in determining which data structure fits their requirements best.…”