A Hybrid Approach to High Availability in Stream Processing Systems

Abstract

Abstract-Stream processing is widely used by today's applications such as financial data analysis and disaster response. In distributed stream processing systems, machine fail-stop events are handled by either active standby or passive standby. However, existing HA schemes have not sufficiently addressed the situation when a machine becomes temporarily unavailable due to data rate spikes, intensive analysis or job sharing, which happens frequently but lasts for short time. It is not clear how well active and passive standby fare against such transient unavailability. In this paper, we first critically examine the suitability of active and passive standby against transient unavailability in a real testbed environment. We find that both approaches have advantages and drawbacks, but neither is ideal to provide fast recovery at low overhead as required to handle transient unavailability. Based on the insights gained, we propose a novel hybrid HA method that switches between active and passive standby modes depending on the occurrence of failure events. It presents a desirable tradeoff that is different from existing HA approaches: low overhead during normal conditions and fast recovery upon transient or permanent failure events. We have implemented our hybrid method and compared it with existing HA designs with comprehensive evaluation. The results show that our hybrid method can reduce two-thirds of the recovery time compared to passive standby and 80% message overhead compared to active standby, allowing applications to enjoy uninterrupted processing without paying a high premium.

Integrating scale out and fault tolerance in stream processing using operator state management

“…However, this requires an average utilisation of less than 50%-active replication is suspended when peak workloads demand all resources. While we assume long-lived failures, Zhang et al [33] focus on transient failures. They combine active replication in the face of failures for fast recovery with passive replication during normal operation.…”

“…While mechanisms for scale out [27,26] and fault tolerance [30,28,33] in stream processing have received considerable attention in the past, it remains an open question how SPSs can scale out while remaining fault tolerant when queries contain stateful operators. Especially with recently popular stream processing models [23,29] that treat operators as black boxes in a data flow graph, users rely on operators that have large amounts of state, which potentially depends on the complete history of previously processed tuples [5].…”

Active Replication at (Almost) No Cost

“…The MapReduce approach requires the operator state to be partitionable, but enables a single operator to scale to a large number of nodes. An approach related, but somehow opposed, to ours is proposed by Zhang et al [18]. In their work passive replication is used when the system is in steady state.…”

A Highly-Efficient Fault Tolerance Method for a Scalable Stream Processing System

“…Therefore, the academic community has put forward a number of fault-tolerant programs based on the SPSs [8-14, 19, 20], most of which are achieved by backing up operator state or the calculation tuples. For instance, Zhang Z et al [19] has reduced both the overhead and latency of the system by mixing the active standby and passive standby, utilizing the ideas of the optimal fault tolerance. Upadhyaya P et al [20] has implemented different fault-tolerant algorithms for different operators so that the recovery time and fault tolerance efficiency can achieve optimal.…”