Managing Tiny Tasks for Data-Parallel, Subsampling Workloads

Kambhampati, Sundeep; Kelley, Jaimie; Stewart, Christopher; Stewart, William C.; Ramnath, Rajiv

doi:10.1109/ic2e.2014.94

Cited by 1 publication

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The case for tiny tasks. Recent work [32,42,44] proposes tiny tasks which run faster and lead to better job completion time when investigating the performance of data analytics jobs. While solutions have been studied to minimize the task launch time [37] and overcome the scheduler overhead [44], tiny tasks hit the performance bottleneck of shuffle when used for large-scale jobs with multiple stages.…”

Section: Related Workmentioning

confidence: 99%

“…Research work highly encourages running a large number of small tasks. Recent work [16,32,[41][42][43] illustrates the benefit of slicing jobs into small tasks: small tasks improve the parallelism, reduce the straggler effect with speculative execution, and speed up end-to-end job completion. Solutions have also been presented to minimize task launch time [37] as well as scheduling overhead [44] for a large number of small tasks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Riffle

Zhang

Cho

Seyfe

et al. 2018

Proceedings of the Thirteenth EuroSys Conference

View full text Add to dashboard Cite

The rapidly growing size of data and complexity of analytics present new challenges for large-scale data processing systems. Modern systems keep data partitions in memory for pipelined operators, and persist data across stages with wide dependencies on disks for fault tolerance. While processing can often scale well by splitting jobs into smaller tasks for better parallelism, all-to-all data transfer-called shuffle operations-become the scaling bottleneck when running many small tasks in multi-stage data analytics jobs. Our key observation is that this bottleneck is due to the superlinear increase in disk I/O operations as data volume increases. We present Riffle, an optimized shuffle service for big-data analytics frameworks that significantly improves I/O efficiency and scales to process petabytes of data. To do so, Riffle efficiently merges fragmented intermediate shuffle files into larger block files, and thus converts small, random disk I/O requests into large, sequential ones. Riffle further improves performance and fault tolerance by mixing both merged and unmerged block files to minimize merge operation overhead. Using Riffle, Facebook production jobs on Spark clusters with over 1,000 executors experience up to a 10x reduction in the number of shuffle I/O requests and 40% improvement in the end-to-end job completion time. CCS CONCEPTS • Computer systems organization → Cloud computing; • Software and its engineering → Ultra-large-scale systems; • Applied computing → Enterprise data management;

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%