Achieving the Performance of All-Bank In-DRAM PIM With Standard Memory Interface: Memory-Computation Decoupling

Abstract

Processing-in-Memory (PIM) has been actively studied to overcome the memory bottleneck by placing computing units near or in memory, especially for efficiently processing low locality dataintensive applications. We can categorize the in-DRAM PIMs depending on how many banks perform the PIM computation by one DRAM command: per-bank and all-bank. The per-bank PIM operates only one bank, delivering low performance but preserving the standard DRAM interface and servicing non-PIM requests during PIM execution. The all-bank PIM operates all banks, achieving high performance but accompanying design issues like thermal and power consumption. We introduce the memory-computation decoupling execution to achieve the ideal all-bank PIM performance while preserving the standard JEDEC DRAM interface, i.e., performing the per-bank execution, thus easily adapted to commercial platforms. We divide the PIM execution into two phases: memory and computation phases. At the memory phase, we read the bank-private operands from a bank and store them in PIM engines' registers bank-by-bank. At the computation phase, we decouple the PIM engine from the memory array and broadcast a bank-shared operand using a standard read/write command to make all banks perform the computation simultaneously, thus reaching the computing throughput of the all-bank PIM. For extending the computation phase, i.e., maximizing all-bank execution opportunity, we introduce a compiler analysis and code generation technique to identify the bank-private and the bank-shared operands. We compared the performance of Level-2/3 BLAS, multi-batch LSTM-based Seq2Seq model, and BERT on our decoupled PIM with commercial computing platforms. In Level-3 BLAS, we achieved speedups of 75.8x, 1.2x, and 4.7x compared to CPU, GPU, and the per-bank PIM and up to 91.4% of the ideal all-bank PIM performance. Furthermore, our decoupled PIM consumed less energy than GPU and the per-bank PIM by 72.0% and 78.4%, but 7.4%, a little more than the ideal all-bank PIM.INDEX TERMS Memory-computation decoupling, in-memory processing, standard memory interface, all-bank execution.

PISA-DMA: Processing-in-Memory Instruction Set Architecture Using DMA

“…Processing-in-Memory (PIM) architectures have been actively studied by placing computing units close to [9], [10], [11], [12], and [13] or inside memory [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26] to overcome the memory bandwidth limitation. PIM can maximize internal memory bandwidth for the computation using bank-level parallelism [14], [15], [17], [18], [22], [23], [24], [25], [26], thus providing high computation performance. For example, the decoupled PIM [26] achieved a speedup of 75.8x and 1.2x over CPU and GPU at the Level-3 BLAS, respectively.…”

“…PIM can maximize internal memory bandwidth for the computation using bank-level parallelism [14], [15], [17], [18], [22], [23], [24], [25], [26], thus providing high computation performance. For example, the decoupled PIM [26] achieved a speedup of 75.8x and 1.2x over CPU and GPU at the Level-3 BLAS, respectively. Samsung FIM [23] achieved a speedup of 11.2x and 3.5x over CPU for memory-bound neural network kernels and applications [27], [29], [32], respectively.…”

“…For example, the latest PIM studies from Samsung [23] and UPMEM [30] separated the PIM memory area from the non-PIM memory to avoid incompatibility with the JEDEC memory standard [31] for supporting all-bank execution. Our recent work and baseline for this research, the decoupled PIM [26] satisfies the standard memory interface. However, its performance is lower than the all-bank PIMs due to its perbank execution.…”

“…It increases the hardware cost and raises the system performance issues, such as making the core very busy and incurring high latency to access uncacheable PIM areas. Recent PIMs use Direct Memory Access (DMA) as the offloading mechanism [22], [25], [26], [30], [32] to resolve the performance issue by transferring opcodes and large-size operands without CPU intervention.…”

“…However, if we express the opcode and operands in a single descriptor together, we can eliminate the opcode descriptors. The elimination allowed us to reduce the total number of descriptors by 25.8%, 26.1%, and 24.9%, thus achieving significant speedups of 1.25x, 1.31x, and 1.29x compared to the baseline PIM [26] in BERT [33], RoBERTa [34], and GPT-2 [35] models, respectively, with a sequence length of 128.…”