Revisiting Memory Errors in Large-Scale Production Data Centers: Analysis and Modeling of New Trends from the Field

Abstract

Abstract-Computing systems use dynamic random-access memory (DRAM) as main memory. As prior works have shown, failures in DRAM devices are an important source of errors in modern servers. To reduce the effects of memory errors, error correcting codes (ECC) have been developed to help detect and correct errors when they occur. In order to develop effective techniques, including new ECC mechanisms, to combat memory errors, it is important to understand the memory reliability trends in modern systems.In this paper, we analyze the memory errors in the entire fleet of servers at Facebook over the course of fourteen months, representing billions of device days. The systems we examine cover a wide range of devices commonly used in modern servers, with DIMMs manufactured by 4 vendors in capacities ranging from 2 GB to 24 GB that use the modern DDR3 communication protocol.We observe several new reliability trends for memory systems that have not been discussed before in literature. We show that (1) memory errors follow a power-law, specifically, a Pareto distribution with decreasing hazard rate, with average error rate exceeding median error rate by around 55 ; (2) non-DRAM memory failures from the memory controller and memory channel cause the majority of errors, and the hardware and software overheads to handle such errors cause a kind of denial of service attack in some servers; (3) using our detailed analysis, we provide the first evidence that more recent DRAM cell fabrication technologies (as indicated by chip density) have substantially higher failure rates, increasing by 1.8 over the previous generation; (4) DIMM architecture decisions affect memory reliability: DIMMs with fewer chips and lower transfer widths have the lowest error rates, likely due to electrical noise reduction; (5) while CPU and memory utilization do not show clear trends with respect to failure rates, workload type can influence failure rate by up to 6.5 , suggesting certain memory access patterns may induce more errors; (6) we develop a model for memory reliability and show how system design choices such as using lower density DIMMs and fewer cores per chip can reduce failure rates of a baseline server by up to 57.7%; and (7) we perform the first implementation and real-system analysis of page offlining at scale, showing that it can reduce memory error rate by 67%, and identify several real-world impediments to the technique.

Workload-Aware DRAM Error Prediction using Machine Learning

“…Although some of these studies utilize workload inherent features, such as the memory reuse time, they are orthogonal to our work. Predictive maintenance and statistical prediction of errors: Considerable research has been done on statistical pre-diction of different types of hardware faults, including DRAM errors, in supercomputers [4], [18], [35], [38], [44], [58], [66], [89]. The majority of these studies proposed different techniques, based either on rules [38] or Machine Learning [66], for prediction of failures that may happen in various hardware components using history of errors.…”

“…Moreover, even though they tried to consider other workload/architecture related factors, this was limited due to the constrained access to only specific features, like percentage of utilized memory, average CPU utilization and hardware characteristics [44]. The joint consideration of more features may reveal new non-linear behaviors that cannot be captured by linear regression models [44] or traditional workload-agnostic statistical models [31]. In addition, all these studies lacked an adequate number of samples because of the rare manifestation of errors for DRAM operating under nominal circuit parameters, which may result in contradictory observations [44], [67].…”

“…The joint consideration of more features may reveal new non-linear behaviors that cannot be captured by linear regression models [44] or traditional workload-agnostic statistical models [31]. In addition, all these studies lacked an adequate number of samples because of the rare manifestation of errors for DRAM operating under nominal circuit parameters, which may result in contradictory observations [44], [67].…”

Robustness of hardware-oriented restricted Boltzmann machines in deep belief networks for reliable processing

“…As a main memory of large-scale computing systems, dynamic random access memories (DRAMs) have been widely used, and have shown a remarkable reliability from the appllication point of view, though mostly owing to the implementation of redundancy and error correction coding (ECC). In modern systems employing DRAMs, however, a growing memory error rate is observed, which depends on the device count; in addition, a significant amount of faults is observed in the memory controllers and transmission channels, [20]. The aforementioned types of faults may affect the neural network itself or the memory in which parameters are temporarily stored.…”

Engineering a Delegatable and Error-Tolerant Algorithm for Counting Small Subgraphs

“…For empirical work and motivation of consideration of errors and other tail events in large-scale infrastructure, cf. Tiwari, Gupta, Gallarno, Rogers, and Maxwell [106], Di Martino, Kalbarczyk, Iyer, Baccanico, Fullop, and Kramer [79], Schroeder, Pinheiro, and Weber [99,100], Meza, Wu, Kumar, and Mutlu [83], Herault and Robert [62], and Barroso, Clidaras, and Hölzle [17].…”