Adaptive Stream Processing using Dynamic Batch Sizing

Abstract

AcknowledgementsMany thanks to Yuan Zhong, Ion Stoica and Scott Shenker for making this thesis possible. Also thanks to David Zats, Shivaram Venkataraman, and Neeraja Yadwadkar for providing feedback on earlier versions of the text.Finally, a very special thanks to both my advisers, Scott Shenker and Ion Stoica, for guiding me through my ups and downs in life and putting up with my idiosyncrasies. 2 AbstractThe need for real-time processing of "big data" has led to the development of frameworks for distributed stream processing in clusters. It is important for such frameworks to be robust against variable operating conditions such as server failures, changes in data ingestion rates, and workload characteristics. To provide fault tolerance and efficient stream processing at scale, recent stream processing frameworks have proposed to treat streaming workloads as a series of batch jobs on small batches of streaming data. However, the robustness of such frameworks against variable operating conditions has not been explored.In this paper, we explore the effect of the size of batches on the performance of streaming workloads. The throughput and end-to-end latency of the system can have complicated relationships with batch sizes, data ingestion rates, variations in available resources, workload characteristics, etc. We propose a simple yet robust control algorithm that automatically adapts batch sizes as the situation necessitates. We show through extensive experiments that this algorithm is powerful enough to ensure system stability and low end-to-end latency for a wide class of workloads, despite large variations in data rates and operating conditions. 3

Towards automatic parameter tuning of stream processing systems

“…Lohrmann et al [37] adaptively adjusted the buffer sizes and performs task chaining according to the QoS constraints. Das et al [23] used dynamic batch sizing for stream processing in Apache Spark to avoid queuing delays as the input data rate changes. Venkataraman et al [50] dynamically adjusted the number of batches that were grouped together for scheduling.…”

Distributed resource management across process boundaries

“…Storm [66] uses Zookeeper [6] to coordinate backpressure across nodes. Das et al [19] propose dynamically adjusting batch sizes to improve latency and throughput. Contrary to these approaches, Wisp's rate limiting approach does not assume knowledge of the full service topology, dynamically computes rate limits based on measured resource utilization instead of static thresholds, is multi-tenant aware, and does not require centralized coordination.…”

Tolerating correlated failures in Massively Parallel Stream Processing Engines

“…foreach Oi do 6 Let p ik be the node in Si that has the largest OF increase δ ik ; 7 P ← P ∪ {p ik }; Si ← Si − {p ik }; 8 usage = N ; 9 if P = ∅ & N > R then return P; 10 while usage < R do 11 Candidates ← ∅; 12 foreach Oi do 13 Let p ik be the node in Si that has the largest OF increase δ ik ;…”

“…Input: The amount of available resources R; Topology T ; Output: Partial replication plan P; 1 Initialize: decompose the complete topology T into sub-topologies: T S1, T S2, ... ; 2 P ← ∅, SA ← ∅, usage ← 0; 3 if R < Number of operators in T then 4 Return P ; 5 foreach Sub-Topology T Si do 6 Ni ← Number of operators in T Si;…”