Design of Processing-“Inside”-Memory Optimized for DRAM Behaviors

Abstract

The computing domain of today's computer systems is moving very fast from arithmetic to data processing as data volumes grow exponentially. As a result, processing-in-memory (PIM) studies have been actively conducted to support the data processing in or near memory devices to address the limited bandwidth and high power consumption due to data movement between CPU/GPU and memory. However, most PIM studies so far have been conducted in a way that the processing units are designed only as an accelerator on the base die of 3D-stacked DRAM, not involved inside memory while not servicing the standard DRAM requests during the PIM execution. Therefore, in this paper, we show how to design and operate the PIM computing units inside DRAM by effectively coordinating with standard DRAM operations while achieving the full computing performance and minimizing the implementation cost. To make our goals, we extend a standard DRAM state diagram to depict the PIM behaviors in the same way as standard DRAM commands are scheduled and operated on the DRAM devices and exploit several levels of parallelism to overlap memory and computing operations. Also, we present how the entire architecture layers from applications to operating systems, memory controllers, and PIM devices should work together for the effective execution by applying our approaches to our experiment platform. In our HBM2-based experimental platform to include 16-cycle MAC (Multiply-and-Add) units and 8-cycle reducers for a matrix-vector multiplication, we achieved 406% and 35.2% faster performance by the all-bank and the per-bank schedulings, respectively, at (1024 × 1024) × (1024 × 1) 8-bit integer matrix-vector multiplication than the execution of only its operand burst reads assuming the external full DRAM bandwidth. It should be noted that the performance of the PIM on a base die of a 3D-stacked memory cannot be better than that provided by the full bandwidth in any case.

Achieving the Performance of All-Bank In-DRAM PIM With Standard Memory Interface: Memory-Computation Decoupling

“…Processing-in-Memory (PIM) has been proposed to alleviate the problem by placing the computing units in the memory [3]- [6], [12]- [21]. The in-bank PIM architecture, typically based on DRAMs, places the PIM engines in the bank peripherals, but the space for the engine implementation is very constrained [3]- [6], [12], [21].…”

“…Processing-in-Memory (PIM) has been proposed to alleviate the problem by placing the computing units in the memory [3]- [6], [12]- [21]. The in-bank PIM architecture, typically based on DRAMs, places the PIM engines in the bank peripherals, but the space for the engine implementation is very constrained [3]- [6], [12], [21]. Also, we should carefully consider the engine operation's power consumption since the high-temperature results in increased unreliable cells, refresh rates, and traffic throttling, leading to performance degradation [22], [23].…”

“…The Level-3 BLAS (i.e., matrix-matrix (MM) multiplication) is more complicated than VM: either a multiplicand row is shared by multiplier columns or a multiplier column is shared by multiplicand rows, as depicted in Figure 1. However, most existing in-bank PIMs are designed for Level-2 BLAS, so they execute Level-3 BLAS by repeating VM multiplications by the number of rows of the multiplicand [3]- [6], [12], [21].…”

“…The recent in-bank PIM can be categorized by whether it targets per-bank execution [4], [21] or all-bank execution [3], [5], [6], [12] depending on how many banks one DRAM command activates for the PIM execution. The per-bank execution triggers only one bank at a time, while the all-bank execution invokes all or multiple banks.…”

“…The number of the commands required for a PIM kernel directly determines the PIM performance. The memory requests for the per-bank PIM execution are much higher than the all-bank one, thus delivering much lower performance [4], [21]. The all-bank PIM achieves high computation throughput by exploiting the bank-level computation parallelism and using the full internal bandwidth.…”