Scalable Data Placement of Data-intensive Services in Geo-distributed Clouds

Abstract

Abstract:The advent of big data analytics and cloud computing technologies has resulted in wide-spread research in finding solutions to the data placement problem, which aims at properly placing the data items into distributed datacenters. Although traditional schemes of uniformly partitioning the data into distributed nodes is the defacto standard for many popular distributed data stores like HDFS or Cassandra, these methods may cause network congestion for data-intensive services, thereby affecting the system throughput. This is because as opposed to MapReduce style workloads, data-intensive services require access to multiple datasets within each transaction. In this paper, we propose a scalable method for performing data placement of data-intensive services into geographically distributed clouds. The proposed algorithm partitions a set of data-items into geodistributed clouds using spectral clustering on hypergraphs. Additionally, our spectral clustering algorithm leverages randomized techniques for obtaining low-rank approximations of the hypergraph matrix, thereby facilitating superior scalability for computation of the spectra of the hypergraph laplacian. Experiments on a real-world trace-based online social network dataset show that the proposed algorithm is effective, efficient, and scalable. Empirically, it is comparable or even better (in certain scenarios) in efficacy on the evaluated metrics, while being up to 10 times faster in running time when compared to state-of-the-art techniques.

UnifyDR: A Generic Framework for Unifying Data and Replica Placement

“…More specifically, any change (small or large) in the system workload would require re-execution of the full pipeline to obtain the placement output. This design decision is in line with almost every existent technique [3]- [5], [16], [17], [43], [49] in the extensive literature on data placement. Thus, making the CDR placement algorithm dynamically adapt to the changes in the system workload is not in the scope of the current work.…”

“…On the other hand, publicly available specialized heuristics for hypergraph partitioning [7] enable graceful scaling of the aforementioned methods to large datasets. Moving further, Atrey et al [3], [5] proposed an algorithm based on spectral clustering of hypergraphs, which portrayed quality similar to the algorithms proposed in [43], however, achieved superior efficiency and scalability owing to the use of randomized eigendecomposition techniques for factorizing the hypergraph laplacian.…”

“…where, the vector W controls the relative importance of the aforementioned hyperedge weight assignment strategies 5 to obtain the final diagonal hyperedge weight matrix W Π of size m × m . To summarize, the hypergraph modeling step produces a hypergraph incidence matrix Π and a hyperedge weight matrix W Π , where the former (Π) models the higher-order interaction between data-items and nodes, while the latter (W Π ) controls the relative importance of different hyperedges.…”

“…• Spectral Clustering (Spectral): obtains a placement using spectral clustering on hypergraphs and achieves superior efficiency by leveraging randomized eigendecomposition methods. Spectral was proposed in [5].…”

“…Recall that as described in Sec. II, the representative stateof-the-art for data placement and replication of data-intensive services into geographically distributed clouds is comprised of the techniques Spectral [5] and Hyper [43].…”