Coherent network interfaces for fine-grain communication

Abstract

Historically, processor accesses to memory-mapped device registers have been marked uncachable to insure their visibility to the device. The ubiquity of snooping cache coherence, however, makes it possible for processors and devices to interact with cachable, coherent memory operations. Using coherence can improve performance by facilitating burst transfers of whole cache blocks and reducing control overheads (e.g., for polling).This paper begins an exploration of network interfaces (NIs) that use coherence---coherent network interfaces (CNIs)---to improve communication performance. We restrict this study to NI/CNIs that reside on coherent memory or I/O buses, to NI/CNIs that are much simpler than processors, and to the performance of fine-grain messaging from user process to user process.Our first contribution is to develop and optimize two mechanisms that CNIs use to communicate with processors. A cachable device register---derived from cachable control registers [39,40]---is a coherent, cachable block of memory used to transfer status, control, or data between a device and a processor. Cachable queues generalize cachable device registers from one cachable, coherent memory block to a contiguous region of cachable, coherent blocks managed as a circular queue.Our second contribution is a taxonomy and comparison of four CNIs with a more conventional NI. Microbenchmark results show that CNIs can improve the round-trip latency and achievable bandwidth of a small 64-byte message by 37% and 125% respectively on the memory bus and 74% and 123% respectively on a coherent I/O bus. Experiments with five macrobenchmarks show that CNIs can improve the performance by 17-53% on the memory bus and 30-88% on the I/O bus.

Eager Data Transfer Mechanism for Reducing Communication Latency in User-Level Network Protocols

“…The work closest to ours is Coherent Network Interface (CNI), which was proposed by Mukherjee et al [5][6][7]. CNI allows a coherent, cacheable memory block implemented as a Cacheable Device Register (CDR) to be shared between the host processor and NI.…”

“…Since both head and tail pointers are needed to determine whether the queue is full or empty, updating of the tail pointer by the host processor and head pointer by the CNI causes invalidations to ping-pong back and forth increasing the traffic on the bus. CNI employs three optimizations based on Lazy pointers, valid bits, and sense reverse to reduce the bus traffic [6]. The idea behind lazy pointers is to remove the host processor's dependency on the head pointer.…”

“…This is done by using a special logic to detect accesses to memory locations for which there are dirty cache blocks, and either allowing data to be accessed directly from cache or implicitly flushing cache blocks before the data can be accessed from the main memory [8,19]. CNI allows data transfer between NI devices registers and cache memory by relying on the underlying cache coherence protocol [5,6]. It effectively uses the bus bandwidth by transferring data in cache-block units and cache invalidations are used as an efficient eventnotification mechanism.…”

Removing the overhead from software-based shared memory

“…This has the advantage of removing the need for asynchronous interrupts in foreign nodes and also allows us to execute the protocol in the requesting processor that most likely would be idle waiting for the data. A further advantage is that the protocol execution is divided between all the processors in the node, not just one processor at a time as suggested in some other proposals, for example by Mukherjee et al [29].…”

“…While other work has proposed elaborate schemes for cutting down on the overhead associated with interrupting and/or polling caused by the asynchronous communication between the agents [5], [29], our implementation has completely eliminated the protocol-agent interactions. In DSZOOM the entire coherence protocol is implemented in the protocol handler running in the requesting processor.…”