Language level checkpointing support for stream processing applications

Abstract

Many streaming applications demand continuous processing of live data with little or no downtime, therefore, making high-availability a crucial operational requirement. Fault tolerance techniques are generally expensive and when directly applied to streaming systems with stringent throughput and latency requirements, they might incur a prohibitive performance overhead. This paper describes a flexible, light-weight fault tolerance solution in the context of the SPADE language and the System S distributed stream processing engine. We devised language extensions so users can define and parameterize checkpoint policies easily. This configurable fault tolerance solution is implemented through code generation in SPADE, which reduces the overall application fault tolerance costs by incurring them only for the parts of the application that require it. In this paper we focus on the overall design of our checkpoint mechanism and we also describe an incremental checkpointing algorithm that is suitable for on-thefly processing of high-rate data streams.

Fault injection-based assessment of partial fault tolerance in stream processing applications

“…Several researchers [4,15,17,21,22,23,30] have described PFT techniques that are applicable to stream processing applications. While many leverage the partial and often strategic employment of fault tolerance techniques to lower the performance loss, no technique can guarantee perfect application output under faulty conditions.…”

“…Aiming at reducing such performance overhead, partial fault tolerance (PFT) techniques [4,17,21,23,30] have been proposed. These techniques assume that data loss and data duplicates are acceptable under faulty conditions.…”

Signal and information processing in mobile cloud computing: Trends and challenges

“…PFT executed partial data duplication and accept some data loss. [24] designed a specialized state serialization methods based on a stream operator checkpoint mechanism. When an operator failed, its upstream operators did not send the data to the operator until the operator was fixed and could generate correct result.…”

From a stream of relational queries to distributed stream processing

“…The operators hosted by PEs implement stream analytics and can be distributed on several compute nodes. System S provides a multiplicity of services, such as fault tolerance mechanisms [26], scheduling and placement mechanisms [19], distributed job management, storage services, and security.…”