Efficient distributed memory management with RDMA and caching

Abstract

Recent advancements in high-performance networking interconnect significantly narrow the performance gap between intra-node and inter-node communications, and open up opportunities for distributed memory platforms to enforce cache coherency among distributed nodes. To this end, we propose GAM, an efficient distributed in-memory platform that provides a directory-based cache coherence protocol over remote direct memory access (RDMA). GAM manages the free memory distributed among multiple nodes to provide a unified memory model, and supports a set of userfriendly APIs for memory operations. To remove writes from critical execution paths, GAM allows a write to be reordered with the following reads and writes, and hence enforces partial store order (PSO) memory consistency. A lightweight logging scheme is designed to provide fault tolerance in GAM. We further build a transaction engine and a distributed hash table (DHT) atop GAM to show the ease-of-use and applicability of the provided APIs. Finally, we conduct an extensive micro benchmark to evaluate the read/write/lock performance of GAM under various workloads, and a macro benchmark against the transaction engine and DHT. The results show the superior performance of GAM over existing distributed memory platforms.

Effective temporal dependence discovery in time series data

“…Hence, the window slicing operator γ is also implemented inside initWindow. Thereafter, for each qualified sliding position of the time window M c .A.W, we compute the causative behavior and allocate the current user to the respective cohort (line [12][13][14]. For dependent behavior measurement, we also first evaluate all the involved time window attributes for each time slice (line 16).…”

“…There have been many frameworks for distributed computation [13,14,15,30,31]. Such frameworks, however, cannot be used for our purpose as the cohort operators cannot be readily mapped into their computation models.…”

“…The intrusive approach is suitable for distributed evaluation. However, existing distributed processing systems [13,14,15,30,31] incur a lot of unnecessary overhead. We hence propose and implement a light-weight distributed architecture for this purpose.…”

The Disruptions of 5G on Data-Driven Technologies and Applications

“…When large amounts of fragmented data are stored across a large number of devices, data have to be processed locally and/or transferred to the cloud for large scale analytics. For local data management and resource sharing over devices, efficient and light data management is required, possibly with some form of distributed shared memory [11].…”

Design and Evaluation of an RDMA-aware Data Shuffling Operator for Parallel Database Systems

“…Gu et al [19] design a remote memory paging system with RDMA that can run applications that do not fit in local memory without modification. Cai et al [8] design GAM, a new distributed in-memory platform that provides cache coherence with RDMA.…”