Machine fault tolerance for reliable datacenter systems

Abstract

Although rare in absolute terms, undetected CPU, memory, and disk errors occur often enough at datacenter scale to significantly affect overall system reliability and availability. In this paper, we propose a new failure model, called Machine Fault Tolerance, and a new abstraction, a replicated writeonce trusted table, to provide improved resilience to these types of failures. Since most machine failures manifest in application server and operating system code, we assume a Byzantine model for those parts of the system. However, by assuming that the hypervisor and network are trustworthy, we are able to reduce the overhead of machine-fault masking to be close to that of non-Byzantine Paxos.

uBFT: Microsecond-scale BFT using Disaggregated Memory

“…With a black-box mechanism to prevent equivocation, one can reduce the number of replicas in BFT to 2𝑓 +1 [15,16,24]. Several BFT systems achieve that using trusted hardware as the black box: attested append-only memory (A2M) [23] uses a trusted log, TrInc [57] and MinBFT [86] use a trusted counter, Hybster [14] uses Intel's SGX, CheapBFT [49] uses FPGAs, and H-MFT [92] uses trusted hypervisors to implement write-once tables. By separating execution from agreement [89], one can reduce the number of execution replicas to 2𝑓 +1, but 3𝑓 +1 replicas are still required for agreement.…”

“…In the slow path-when there are failures or slowness in the network-uBFT uses a novel protocol that combines digital signatures with judicious use of a trusted computing base. The trusted computing base in uBFT is non-tailored and small: rather than trusted enclaves with arbitrary logic such as Intel's SGX [26] or trusted hypervisors [92]-which have large attack surfaces due to their complexity [27,34]-uBFT relies solely on disaggregated memory, a technology increasingly present in data centers due to the availability of RDMA [83] today and CXL [28] in a few years. The key mechanism from disaggregated memory we leverage in uBFT are single-writer regions (regions of memory that can be written by one designated host and can be read by others), implemented in hardware through access permissions.…”

“…To get microsecond-scale latency, BFT protocols need to avoid expensive public-key cryptography and reduce communication rounds in the common path-and doing so has typically required increasing rather than decreasing the number of replicas [1,51,62]. Meanwhile, decreasing the number of replicas has usually required unbounded memory, sophisticated, or tailored trusted computing bases such as append-only-memory [23], SGX [14], TrInc [57], or reliable hypervisors [92]. Limiting the amount of memory is a significant challenge in the design of uBFT, as the standard technique to handle Byzantine behavior in systems with 2𝑓 +1 replicas requires storing all messages received, leading to long message histories [4,86], which consume unbounded memory.…”