Scotty: Efficient Window Aggregation for Out-of-Order Stream Processing

Abstract

Computing aggregates over windows is at the core of virtually every stream processing job. Typical stream processing applications involve overlapping windows and, therefore, cause redundant computations. Several techniques prevent this redundancy by sharing partial aggregates among windows. However, these techniques do not support out-of-order processing and session windows. Out-of-order processing is a key requirement to deal with delayed tuples in case of source failures such as temporary sensor outages. Session windows are widely used to separate different periods of user activity from each other. In this paper, we present Scotty, a high throughput operator for window discretization and aggregation. Scotty splits streams into non-overlapping slices and computes partial aggregates per slice. These partial aggregates are shared among all concurrent queries with arbitrary combinations of tumbling, sliding, and session windows. Scotty introduces the first slicing technique which (1) enables stream slicing for session windows in addition to tumbling and sliding windows and (2) processes out-of-order tuples efficiently. Our technique is generally applicable to a broad group of dataflow systems which use a unified batch and stream processing model. Our experiments show that we achieve a throughput an order of magnitude higher than alternative stateof-the-art solutions.

Optimal and general out-of-order sliding-window aggregation

“…The insert(t, v) operation supports the case where t is already in the window, so it works with pre-aggregation schemes such as window panes [23], paired windows [22], cutty windows [13], or Scotty [30]. For instance, for a 5-hour sliding window that advances in 1-minute increments, the logical times can be rounded to minutes, leading to more cases where t is already in the window.…”

“…All of the following papers focus on sharing over streams with the same aggregation operator, e.g., monoid (S, ⊗, 1). The Scotty algorithm supports sliding-window aggregation over out-oforder streams, while sharing windows with both different sizes and slice granularities [30]. For instance, Scotty might share a window of size 60 minutes and granularity 3 minutes with a session window whose gap timeout is set to 5 minutes.…”

Stream Window Aggregation Semantics and Optimization

“…Another future direction is to extend slicing capabilities beyond deterministic windows (if possible) and cover cases of fully data-driven windows without FIFO guarantees such as ADWIN (Bifet and Gavalda 2007). Finally, general pre-aggregation data structures have to employ the notion of out-of-orderness (Traub et al 2018). Currently, with existing out-of-the-box solution such as FlatFat it is not possible to retract already evaluated window aggregates, thus, making it impossible to use for systems like Beam and Flink with out-of-order logic.…”

Generating Reproducible Out-of-Order Data Streams

“…Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s [3,8,10,11]. To evaluate these techniques and the impact of out-of-order handling on the system performance, it is necessary to modify real-world data sets in order to meet specific data characteristics such as out-of-orderness and the addition of delay in event arrivals.…”

“…In this paper, we present an out-of-order stream data generator, which enables the reproducible modification of out-of-order characteristics of arbitrary input data sets. This tool was already used in recent work on efficient window aggregation with General Stream Slicing [10,11] and is available in a public repository. 1 The data generator supports benchmark developers in two aspects.…”