Achieving High-Throughput State Machine Replication in Multi-core Systems

Abstract

Abstract-State machine replication is becoming an increasingly popular technique among online services to ensure faulttolerance using commodity hardware. This has led to a renewed interest in its throughput, as these services have typically a large number of users. Recent work has shown how to improve throughput of the ordering phase by improving the replication protocol, using techniques like Ring-topologies, IP multicast, and rotating leaders. The resulting systems, when deployed in modern fast networks, achieve unprecedented levels of throughput. But these systems are increasingly becoming limited by the CPU of the replicas, especially with small client requests. However, the problem is not lack of performance of the CPUs, but instead the inability of typical implementations to effectively use the multiple cores of modern multi-core CPUs. In this work, we show how to architect a replicated state machine whose performance scales with the number of cores. We do so by applying several good practices of concurrent programming to the specific case of state machine replication, including staged execution, workload partitioning, actors, and non-blocking data structures. We describe and test a Java prototype of our architecture, based on the Paxos protocol. With a workload consisting of small requests, we achieve a 6 times improvement in throughput using 8 cores. More generally, in all our experiments we have consistently reached the limits of network subsystem by using up to 12 cores, and do not observe any degradation when using up to 24 cores. Furthermore, the profiling results of our implementation show that even at peak throughput contention between threads is minimal, suggesting that the throughput would continue scaling given a faster network.

State Machine Replication for the Masses with BFT-SMART

“…Nonetheless, this result is consistent with recent reports about PBFT performance (e.g., [8]). JPaxos displayed a performance lower than what is reported in [16] (around 100 Kops/sec). Since we are using the same type of network, the only reason for that is their use of machines with more hardware threads than ours (24 vs. 16).…”

“…Comparison with others: We compared BFT-SMART against some representative SMR systems considering the 0/0 benchmark. More precisely, we compared BFT-SMART (both in BFT and CFT setups) with PBFT [2], UpRight [4] and JPaxos [16] (a modern multi-core CFT replication library). All systems were downloaded from the internet 3 in October 2013, installed and configured to mimic the setup used in their respective papers.…”

“…This is significantly less than what was used in similar systems: PBFT [2] contains 20K lines of C code and UpRight [4] contains 22K lines of Java code. Even JPaxos [16], the most complete open-source CFT replication library we are aware of, contains more than 22K lines of Java code.…”

On the Design of Practical Fault-Tolerant SDN Controllers

“…We further demonstrated that the state-of-the-art distributed systems techniques (e.g., [14], [23], [5], [7]) we used to optimize throughput has resulted in a significant performance improvement over the values reported in the Onix paper for its consistent data store. To further increase the performance and scalability of our solution, in this work we incorporate a cache in the controllers to avoid accessing the shared data store on reads.…”

Scalable State-Machine Replication

“…State machine replication is a well-known approach to replication and has been extensively studied (e.g., (KAPRITSOS et al, 2012;KOTLA;DAHLIN, 2004;LAMPORT, 1978;SANTOS;SCHIPER, 2013;SCHNEIDER, 1990)). State machine replication requires replicas to execute commands deterministically, which implies sequential execution.…”