The BTWorld use case for big data analytics: Description, MapReduce logical workflow, and empirical evaluation

Abstract

Abstract-The commoditization of big data analytics, that is, the deployment, tuning, and future development of big data processing platforms such as MapReduce, relies on a thorough understanding of relevant use cases and workloads. In this work we propose BTWorld, a use case for time-based big data analytics that is representative for processing data collected periodically from a global-scale distributed system. BTWorld enables a datadriven approach to understanding the evolution of BitTorrent, a global file-sharing network that has over 100 million users and accounts for a third of today's upstream traffic. We describe for this use case the analyst questions and the structure of a multi-terabyte data set. We design a MapReduce-based logical workflow, which includes three levels of data dependencyinter-query, inter-job, and intra-job-and a query diversity that make the BTWorld use case challenging for today's big data processing tools; the workflow can be instantiated in various ways in the MapReduce stack. Last, we instantiate this complex workflow using Pig-Hadoop-HDFS and evaluate the use case empirically. Our MapReduce use case has challenging features: small (kilobytes) to large (250 MB) data sizes per observed item, excellent (10 −6 ) and very poor (10 2 ) selectivity, and short (seconds) to long (hours) job duration.

Tyrex: Size-Based Resource Allocation in MapReduce Frameworks

“…BTWORLD is a complex and very challenging MapReduce-based logical workflow which we designed to observe the evolution of the global-scale peerto-peer system BitTorrent [18]. The workflow consists of 26 MapReduce jobs with different resource bottlenecks (CPU, memory, or disk) and is used for processing monitoring data collected periodically from the BitTorrent system.…”

“…In our experience with processing monitoring data from the BitTorrent global network using a MapReduce-based logical workflow [18], we have found that 15% of the jobs account for 80% of the total load, and that 65% of the jobs complete in a minute. Similarly, several studies on the performance of modern production clusters in 1 Inspired by the dinosaur Tyrannosaurus rex, known for its long, heavy tail.…”

V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows

“…At the same time, we surpass our own previous work by a factor of 15 [2]. Most importantly though, the problem we are addressing in this paper concerns the global BitTorrent network, which is of an unprecedented scale in peer-to-peer research.…”

“…Starting with 2009, in our BTWorld project [1] we have conducted a longitudinal experiment in observing the global BitTorrent network during which we have collected over 15 TB of operational data. Although we have created a MapReduce-based logical workflow to extract insightful knowledge about the evolution of the BitTorrent network [2], the vicissitude of processing our BitTorrent data, that is the combination between large volume of data and the complexity of the processing workflow, has prevented us until now to gather useful insights. To address this problem, in this work we demonstrate the scaling of the BTWorld workflow and process an order of magnitude more data than in our previous attempt [2].…”

“…2) We design a multi-level optimization cycle for tuning MapReduce-based software stacks in order to accommodate complex workflows. 3) We present the scaling of the BTWorld workflow to 15 times the size of the data sets of our previous results [2].…”