Computing marginals using MapReduce

Abstract

We consider the problem of computing the data-cube marginals of a fixed order k (i.e., all marginals that aggregate over k dimensions), using a single round of MapReduce. The focus is on the relationship between the reducer size (number of inputs allowed at a single reducer) and the replication rate (number of reducers to which an input is sent). We show that the replication rate is minimized when the reducers receive all the inputs necessary to compute one marginal of higher order. That observation lets us view the problem as one of covering sets of k dimensions with sets of a larger size m, a problem that has been studied under the name "covering numbers." We offer a number of constructions that, for different values of k and m meet or come close to yielding the minimum possible replication rate for a given reducer size. Background MarginalsConsider an n-dimensional data cube [11] and the computation of its marginals by MapReduce. A marginal of a data cube is the aggregation of the data in all those tuples that have fixed values in a subset of the dimensions of the cube. We shall assume this aggregation is the sum, but the exact nature of the aggregation is unimportant in what follows. Marginals can be represented by a list whose elements correspond to each dimension, in order. If the value in a dimension is fixed, then the fixed value represents the dimension. If the dimension is aggregated, then there is a * for that dimension. The number of dimensions over which we aggregate is the order of the marginal. Example 1.1. Suppose n = 5, and the data cube is a relation DataCube(D1,D2,D3,D4,D5,V). Here, D1 through D5 are the dimensions, and V is the value that is aggregated. SELECT SUM(V)FROM