HALCONE : A Hardware-Level Timestamp-based Cache Coherence Scheme for Multi-GPU systems

Abstract

While multi-GPU (MGPU) systems are extremely popular for compute-intensive workloads, several inefficiencies in the memory hierarchy and data movement result in a waste of GPU resources and difficulties in programming MGPU systems. First, due to the lack of hardware-level coherence, the MGPU programming model requires the programmer to replicate and repeatedly transfer data between the GPUsâ Ȃ Ź memory. This leads to inefficient use of precious GPU memory. Second, to maintain coherency across an MGPU system, transferring data using low-bandwidth and high-latency off-chip links leads to degradation in system performance. Third, since the programmer needs to manually maintain data coherence, the programming of an MGPU system to maximize its throughput is extremely challenging. To address the above issues, we propose a novel lightweight timestampbased coherence protocol, HALCONE , for MGPU systems and modify the memory hierarchy of the GPUs to support physically shared memory. HALCONE replaces the Compute Unit (CU) level logical time counters with cache level logical time counters to reduce coherence traffic. Furthermore, HALCONE introduces a novel timestamp storage unit (TSU) with no additional performance overhead in the main memory to perform coherence actions. Our proposed HAL-CONE protocol maintains the data coherence in the memory hierarchy of the MGPU with minimal performance overhead (less than 1%). Using a set of standard MGPU benchmarks, we observe that a 4-GPU MGPU system with shared memory and HALCONE performs, on average, 4.6× and 3× better than a 4-GPU MGPU system with existing RDMA and with the recently proposed HMG coherence protocol, respectively. We demonstrate the scalability of HALCONE using different GPU counts (2, 4, 8, and 16) and different CU counts (32, 48, and 64 CUs per GPU) for 11 standard benchmarks. Broadly, HALCONE scales well with both GPU count and CU count. Furthermore, we stress test our HALCONE protocol using a custom synthetic benchmark suite to evaluate its impact on the overall performance. When running our synthetic benchmark suite, the HALCONE protocol slows down the execution time by only 16.8% in the worst case.

MGPU-TSM: A Multi-GPU System with Truly Shared Memory

“…In the MGPU-TSM system, different CUs within and across GPUs can access the same memory location. Hence, we need a low-overhead scalable cache coherency and memory consistency model to maintain accuracy such as HAL-CONE [7]. Traditional snooping-based or directory-based coherency protocols, such as MESI and MOESI, can lead to large inter-GPU and intra-GPU communication latencies [12].…”