High throughput reliable message dissemination

Abstract

We consider applications that require high rate, reliable message dissemination in a many-to-many environment. Examples of such applications include stock market centers and synchronized server clusters. As network capacity increases, the achievable throughput of messaging applications becomes bounded by processing times rather than communication speed. To reduce processing times we suggest the use of message aggregation. We consider performing message aggregation at either the sender, a message-server, or a network switch. The performance of each of these methods in terms of throughput and delay is analytically evaluated and compared against that of a naive implementation that does not perform message aggregation. We show that in typical real-world messaging applications, performing message aggregation can increase throughput by order of magnitude.We base our results on experiments that have been conducted using various operating systems running on different hardware platforms. Our results indicate that the achievable throughput of messaging applications is determined by the number of packets-per-second, rather than bytes-per-second, a receiver or a transmitter should handle.

Tuning Paxos for High-Throughput with Batching and Pipelining

“…Batching is an example of message aggregation, which has been previously studied as a way of reducing the fixed per-packet overhead by spreading it over a large number of data or messages, see [2,3,5,6]. It is also widely deployed, with TCP's Nagle algorithm [10] being a notable example.…”

“…Batching has been studied as a general technique by [3] and [5]. In [3] the authors present a detailed analytical study, quantifying the effects of batching on reliable message transmission protocols.…”

“…In [3] the authors present a detailed analytical study, quantifying the effects of batching on reliable message transmission protocols. One of the main difficulties in batching is deciding when to stop waiting for additional data and form a batch.…”

“…These include interrupt handling and context switching as a result of reading and writing data to the network card, allocating buffers, updating the replicated log and the internal data structures, and executing the protocol logic. In [3], the authors show that the fixed costs of sending a packet over a Ethernet network are dominant for small packet sizes, and that for larger packets the total processing time grows significantly slower than the packet size. In the case of Paxos, the fixed costs of an instance are an even larger fraction of the total costs because, in addition to processing individual messages, processes also have to execute the ordering algorithm.…”

“…Therefore, a batching policy must strike a balance between creating large batches (to improve throughput) and deciding when to stop waiting for additional requests and send the batch (to keep latency within acceptable bounds). This problem has been studied in the general context of communication protocols by [2,3,5]. In the rest of the paper, we study it in the context of Paxos, and analyze its interaction with the pipelining optimization.…”