Understanding the idiosyncrasies of real persistent memory

Abstract

High capacity persistent memory (PMEM) is finally commercially available in the form of Intel's Optane DC Persistent Memory Module (DCPMM). Researchers have raced to evaluate and understand the performance of DCPMM itself as well as systems and applications designed to leverage PMEM resulting from over a decade of research. Early evaluations of DCPMM show that its behavior is more nuanced and idiosyncratic than previously thought. Several assumptions made about its performance that guided the design of PMEM-enabled systems have been shown to be incorrect. Unfortunately, several peculiar performance characteristics of DCPMM are related to the memory technology (3D-XPoint) used and its internal architecture. It is expected that other technologies (such as STT-RAM, memristor, ReRAM, NVDIMM), with highly variable characteristics, will be commercially shipped as PMEM in the near future. Current evaluation studies fail to understand and categorize the idiosyncratic behavior of PMEM; i.e., how do the peculiarities of DCPMM related to other classes of PMEM. Clearly, there is a need for a study which can guide the design of systems and is agnostic to PMEM technology and internal architecture. In this paper, we first list and categorize the idiosyncratic behavior of PMEM by performing targeted experiments with our proposed PMIdioBench benchmark suite on a real DCPMM platform. Next, we conduct detailed studies to guide the design of storage systems, considering generic PMEM characteristics. The first study guides data placement on NUMA systems with PMEM while the second study guides the design of lock-free data structures, for both eADR- and ADR-enabled PMEM systems. Our results are often counter-intuitive and highlight the challenges of system design with PMEM.

GPM: leveraging persistent memory from a GPU

“…Impact of upcoming technologies on GPM's design: Intel recently announced eADR (enhanced ADR) feature in their future server processors [36]. This hardware feature, along with 2 nd generation Optane NVM will drain the entire contents of CPU caches to PM on power failures [3,27,36,39]. This feature will obviate the need to flush cache blocks from CPU's caches in order to guarantee persistence in future processors.…”

Dynamic Page Placement on Real Persistent Memory Systems

“…This cache also serves as a write-combining buffer for adjacent stores. Due to the granularity mismatch between DDR4 and XPLines, random stores incur in costly readmodify-write cycles [14]. Similar to SSDs, DCPMM uses logical addressing for minimizing wear-leveling, leveraging an internal address indirection table [39].…”

“…Since the availability of DCPMM, there are some initial studies of data placement in real DRAM+DCPMM systems. Some authors studied the system performance of DCPMM [39], [56], [32], [14], and others have manually modified data placement in either mini or real applications and evaluated the performance benefits in DRAM+DCPMM systems [37], [52], [43], [29], [51], [58]. Finally, recent profiling-based proposals to guide or automate static data placement have already been implemented and evaluated with real DCPMM [9], [38].…”

“…First, these proposals are designed based on conjectural assumptions about the performance of then future PMbased systems. Furthermore, their experimental evaluation relies on simulation or emulation methods that fail to capture performance intricacies that are only noticeable with the real technology [14].…”

LightPC

“…Instead, we can leverage a persistent memory module (PMEM) supporting 10× higher storage capacity with excellent non-volatile capabilities, offered by 3D-Xpoint [13,14], which is a variation of phase change memory (PRAM) [15][16][17]. However, modern systems employing PMEM unfortunately undergo unexpected performance and latency variation [18][19][20]. For example, [18] reports that the processor latency with PMEM is non-deterministic and significantly varies, which is 3× worse than a legacy system using DRAM.…”