Implementing reliable event streams in large systems via distributed data flows and recursive delegation

Abstract

Strong reliability properties, such as state machine replication and virtual synchrony, are hard to implement in a scalable manner. They are typically expressed in terms of global membership views. However, global membership is non-scalable. We propose a new way of modeling protocols that does not rely on global membership. Our approach is based on the concept of a distributed data f ow, a set of events distributed in space and time. We model protocols as networks of such f ows, constructed through recursive delegation. The resulting system uses multiple small membership services instead of a single global service while still supporting stronger properties. This paper focuses on the theoretical model and its base properties; in particular, on the concept of monotonic aggregation. We present a high-level architecture overview and initial performance results.

Distributed data flow language for multi-party protocols

“…By default, aggregation has certain important properties that make it possible to build strong semantics. The MA concept is discussed at length in our past work [25]. Due to limited space, here we limit ourselves to pointing out that MA could be translated to a simple token ring that self-organizes using an external membership service, but it can also be translated into a scalable, hierarchical architecture [25].…”

“…The MA concept is discussed at length in our past work [25]. Due to limited space, here we limit ourselves to pointing out that MA could be translated to a simple token ring that self-organizes using an external membership service, but it can also be translated into a scalable, hierarchical architecture [25]. MA is a very simple, lightweight and scalable abstraction that can be composed to express a great variety of complex DMPs with strong semantics such as atomic delivery or consensus.…”

“…This is done by intersecting the respective set values (a message is "stable" if it's been received everywhere in the group; the interpretation of everywhere in our model has been discussed in our past work [25]). In our simulations [25], token ring aggregation at a few rounds/s made commit and abort decisions for thousands of transactions/s by using set arithmetic. We've also used this approach successfully in our hand-coded high-performance scalable multicast platforms.…”

“…Data flows in the sense of asynchronous, massively parallel, pipelined processing, have a long tradition in areas such as VLSI/DBMS, and aggregation has been extensively studied in the context of sensor networks; a discussion of the relevant prior work can be found elsewhere [24], [25]. The key difference is that data flows in those systems are not distributed in the same sense as in our model: they are point-to-point event streams, transformations on them are local, and they lack the sort of strong semantics needed to express DMPs.…”

“…space, this paper discusses only aggregation and batching. The DF concept, and the underlying theory for how strong reliability properties can be expressed through monotonic aggregation, have been discussed in our past work [25], and a brief discussion of recursion can be found in our tech report [24]. The latter includes a language grammar and more protocol examples (omitted here for brevity).…”