Low-Overhead Fault Tolerance for High-Throughput Data Processing Systems

Abstract

The MapReduce programming paradigm proved to be a useful approach for building highly scalable data processing systems. One important reason for its success is simplicity, including the fault tolerance mechanisms. However, this simplicity comes at a price: efficiency. MapReduce's fault tolerance scheme stores too much intermediate information on disk. This inefficiency negatively affects job completion time. Furthermore, this inefficiency in particular forbids the application of MapReduce in near real-time scenarios where jobs need to produce results quickly. In this paper, we discuss an alternative fault tolerance scheme that is inspired by virtual synchrony. The key feature of our approach is a low-overhead deterministic execution. Deterministic execution reduces the amount of persistently stored information. In addition, because persisting intermediate results are no longer required for fault tolerance, we use more efficient communication techniques that considerably improve job completion time and throughput. Our contribution is twofold: (i) we enable the use of MapReduce for jobs ranging from seconds to a few tens of seconds, satisfying these deadlines even in the case of failures; (ii) we considerably reduce the fault tolerance overhead and as such the overhead of MapReduce in general. Our modifications are transparent to the application.

Adaptive online scheduling in storm

“…This kind of stratification has been investigated in [19]. Recent works on streaming MapReduce [21,20,11] also focus on the possibility to model any computation as a sequence of alternated map and reduce stages, supporting the idea that a large class of computations can be structured as a sequence of consecutive stages where events always flow from previous to subsequent stages.…”

“…Other works [21,20,11] try to bridge the gap between continuous queries and MapReduce paradigm by proposing a stream based version of the MapReduce approach [3] where events uninterruptedly flow among the map and reduce stages of a certain computation without incurring in the delays typical of batch oriented solutions.…”

Elastic Scaling of a High-Throughput Content-Based Publish/Subscribe Engine

“…E-STREAMHUB supports full elasticity, i.e., scale out and in, while optimizing the overall system utilization, and without requiring specific application support. Its underlying runtime engine can support both passive [26] and active [25] replication.…”

“…Each slice can be supported by multiple cores operating in parallel when the processing is stateless or requires only an R lock. STREAMMINE3G supports dependability through passive [26] or active [25] slice replication. This requires no modification to the application but the evaluation of these mechanisms is out of the scope of the present paper.…”

Active Replication at (Almost) No Cost

“…If nodes rely on state, this state needs to be keep safe for when failures occur. In a previous work, we have detailed an approach for passive replication for ESP systems that apply the MapReduce approach [13]. However, as detailed in Section II, traditional passive replication has a low resource overhead, but requires a long recovery phase after a failure as both the checkpoint needs to be restored and events from the log need to be replayed and reprocessed.…”