Parallelizing the Data Cube

Abstract

This paper presents a general methodology for the e cient parallelization of existing data cube construction algorithms. We describe two di erent partitioning strategies, one for top-down and one for bottom-up cube algorithms. Both partitioning strategies assign subcubes to individual processors in such a way that the loads assigned to the processors are balanced. Our methods reduce inter-processor communication overhead by partitioning the load in advance instead of computing each individual group-by in parallel as is done in previous parallel approaches. In fact, after the initial load distribution phase, each processor can compute its assigned subcube without any communication with the other processors. Our methods enable code reuse by permitting the use of existing sequential (external memory) data cube algorithms for the subcube computations on each processor. This supports the transfer of optimized sequential data cube code to a parallel setting.The bottom-up partitioning strategy balances the number of single attribute external memory sorts made by each processor. The top-down strategy partitions a weighted tree in which weights re ect algorithm speci c cost measures like estimated group-by sizes. Both partitioning approaches can beimplement e d o n a n y shared disk type parallel machine composed of p processors connected via an interconnection fabric and with access to a shared parallel disk array.We h a ve implemented our parallel top-down data cube construction method in C++ with the MPI message passing library for communication and the LEDA library for the required graph algorithms. We tested our code on an eight processor cluster, using a variety of di erent data sets with a range of sizes, dimensions, density, and skew. The tests show that our partitioning strategies generate a close to optimal load balance between processors. The actual run times observed show an optimal speedup of p.2

Efficient and Self-Balanced ROLLUP Aggregates for Large-Scale Data Summarization

“…They are divided into two main groups: work partitioning [11], [12] and data partitioning [13], [14]. In work partitioning, each processor (or node) of the cluster computes aggregates for a set of one or many views independently.…”

A distributed tree data structure for real-time OLAP on cloud architectures

“…Building the data cube can be a massive computational task, and significant research has been published on sequential and parallel data cube construction methods (e.g. [4], [5], [3], [6], [7], [8]). However, the traditional static data cube approach has several disadvantages.…”

Substantiating Anomalies In Wireless Networks Using Group Outlier Scores

“…that together form a key. Measures are typically numeric elements like packet The pre-computation of the different views of a data cube (i.e., the forming of aggregates for every combination of GROUP BY attributes) is critical to improving the response time of the queries [18]. However, in many cases not all views are needed for decision making, therefore it is advantageous to use only selected views.…”