Zyzzyva

Abstract

A longstanding vision in distributed systems is to build reliable systems from unreliable components. An enticing formulation of this vision is Byzantine Fault-Tolerant (BFT) state machine replication, in which a group of servers collectively act as a correct server even if some of the servers misbehave or malfunction in arbitrary (“Byzantine”) ways. Despite this promise, practitioners hesitate to deploy BFT systems, at least partly because of the perception that BFT must impose high overheads. In this article, we present Zyzzyva, a protocol that uses speculation to reduce the cost of BFT replication. In Zyzzyva, replicas reply to a client's request without first running an expensive three-phase commit protocol to agree on the order to process requests. Instead, they optimistically adopt the order proposed by a primary server, process the request, and reply immediately to the client. If the primary is faulty, replicas can become temporarily inconsistent with one another, but clients detect inconsistencies, help correct replicas converge on a single total ordering of requests, and only rely on responses that are consistent with this total order. This approach allows Zyzzyva to reduce replication overheads to near their theoretical minima and to achieve throughputs of tens of thousands of requests per second, making BFT replication practical for a broad range of demanding services.

The Next 700 BFT Protocols

“…It makes Aliph the first BFT protocol with a number of MAC operations at the bottleneck server that tends to 1 in the absence of asynchrony/failures. This contradicts the claim that the lower bound is 2 [Kotla et al 2010]. Interestingly, each of Quorum and Chain could be developed independently and required less than 35% of the code needed to develop state-of-the-art BFT protocols.…”

“…We describe AZyzzyva, a full-fledged BFT protocol that mimics Zyzzyva [Kotla et al 2010] in its "common case" (i.e., when there are no link or server failures). In "other cases," AZyzzyva relies on Backup, an Abstract implementation with strong progress guarantees that can be implemented on top of any existing BFT protocol.…”

“…As mentioned before, AZyzzyva is a BFT protocol that mimics Zyzzyva [Kotla et al 2010]. Zyzzyva requires 3 f + 1 replicas.…”

“…This second phase thus aims at handling the case with link/server/client failures ("worst case"). Roughly, this phase (that AZyzzyva avoids to mimic) consists of considerable modifications to PBFT [Castro and Liskov 2002], which arise from the "profound effects" [Kotla et al 2010] that the Zyzzyva "common case" optimizations have on PBFT's "worst case. "…”

A History of the Virtual Synchrony Replication Model

“…Now, not all BFT work is aimed at code for imaginary, unrealizable computing devices. Here in the SOSP community we would tend to point to papers like the PRACTI work by Castro and Liskov [33], the so-called CASD (delta-T) atomic broadcast protocol of Cristian, Aghili, Strong and Dolev [49], or the Zyzzyva paper by Kotla and Alvisi [34], in which BFT is integrated with a gossip protocol [51] that can be used in real networks. But these results are not typical of the main body of BFT work, which really is focused on how to write code for an imaginary kind of data center that embodies unrealizable hardware.…”