Coded Data Rebalancing: Fundamental Limits and Constructions

Abstract

Distributed databases often suffer unequal distribution of data among storage nodes, which is known as 'data skew'. Data skew arises from a number of causes such as removal of existing storage nodes and addition of new empty nodes to the database. Data skew leads to performance degradations and necessitates 'rebalancing' at regular intervals to reduce the amount of skew. We define an r-balanced distributed database as a distributed database in which the storage across the nodes has uniform size, and each bit of the data is replicated in r distinct storage nodes. We consider the problem of designing such balanced databases along with associated rebalancing schemes which maintain the r-balanced property under node removal and addition operations. We present a class of r-balanced databases (parameterized by the number of storage nodes) which have the property of structural invariance, i.e., the databases designed for different number of storage nodes have the same structure. For this class of r-balanced databases, we present rebalancing schemes which use coded transmissions between storage nodes, and characterize their communication loads under node addition and removal. We show that the communication cost incurred to rebalance our distributed database for node addition and removal is optimal, i.e., it achieves the minimum possible cost among all possible balanced distributed databases and rebalancing schemes.

An Umbrella Converse for Data Exchange: Applied to Caching, Computing, and Shuffling

“…The above data exchange problem models a number of cache-enabled multi-receiver communication problems studied recently in the coding theory community, including Coded Caching [1], Coded Distributed Computing [2,3], Coded Data Shuffling [4][5][6], and Coded Data Rebalancing [7]. In [8], a special case of our general problem here was considered in the name of cooperative data exchange, where the goal was to reach a state in which all nodes have all the data in the system.…”

“…available exclusively in Q ∪ 0 Symmetrization over appropriately chosen demand vectors: Choosing the same special set of demand vectors D as in (7) and averaging the above lower bound over the demand vectors in D similar to the proof of Theorem 2, we obtain a bound similar to (8):…”

An Umbrella Converse for Data Exchange: Applied to Caching, Computing, and Shuffling

“…• For the problem of Coded Data Rebalancing, introduced in [4], we show a new converse for data rebalancing under multiple node failures. (Section V) • We present a new converse for a generic coded caching setting with multiple transmitters, receivers, and multilevel cache sizes.…”

“…Natarajan is with the Department of Electrical Engineering, Indian Institute of Technology Hyderabad, email: lakshminatarajan@iith.ac.in. recently in the coding theory community, including Coded Caching [1], Coded Distributed Computing [2], Coded Data Shuffling [3], and Coded Data Rebalancing [4].…”

“…The present authors formally introduced this problem in [4], and designed balanced distributed databases and associated rebalancing schemes for handling single-node removals from and single-node additions to these databases. These rebalancing schemes have provably minimum possible communication cost.…”