Advanced join strategies for large-scale distributed computation

Abstract

Companies providing cloud-scale data services have increasing needs to store and analyze massive data sets (e.g., search logs, click streams, and web graph data). For cost and performance reasons, processing is typically done on large clusters of thousands of commodity machines by using high level scripting languages. In the recent past, there has been significant progress in adapting well-known techniques from traditional relational DBMSs to this new scenario. However, important challenges remain open. In this paper we study the very common join operation, discuss some unique challenges in the large-scale distributed scenario, and explain how to efficiently and robustly process joins in a distributed way. Specifically, we introduce novel execution strategies that leverage opportunities not available in centralized scenarios, and others that robustly handle data skew. We report experimental validations of our approaches on Scope production clusters, which power the Applications and Services Group at Microsoft.

A Coflow-Based Co-Optimization Framework for High-Performance Data Analytics

“…Within the scope of current study in the filed of data management (e.g., [4], [6]), the schedule plan SP 2 will be considered as an optimal solution and chosen by underlying systems, because it transfers less data than other two approaches.…”

“…To data, large number of techniques have been proposed to handle data skew in join executions [5], [6], [11], [16]. Among them, we have chosen a very efficient method, partial duplication [11], in our implementations.…”

“…In fact, these operations have been extensively studied in the field of data management, and large number of methods have been proposed to improve their performance. For example, for distributed join executions, current research focuses on the challenge on how to efficiently move data, either in the presence of different join workloads (e.g., skew) or different computing platforms (e.g., clusters and Cloud) or both [4], [5], [6], [11], [16], [20]. Their main target is either to reduce network traffic or to improve load-balancing or both, so as to balance computations and improve network communication time.…”

“…Based on this, various advanced approaches [5] and strategies [6] have been proposed in the data management domain. Their philosophy is generally to move small data chunks instead of large data chunks during their executions.…”

A Theoretical and Experimental Comparison of Filter-Based Equijoins in MapReduce

“…Join processing in MapReduce has become a hot research topic in recent years [8,3,16,2,22,30,11]. Many studies have been carried out to evaluate join queries and analyze large datasets in a MapReduce environment.…”