System Architecture and Software Stack for GDDR6-AiM

Kwon, Yongkee; Vladimir, Kornijcuk; Kim, Nahsung; Shin, Woojae; Won, Jong-Soon; Lee, Min-Kyu; Joo, Hyunha; Choi, Haerang; Kim, Guhyun; An, Byeong-Ju; Kim, Jeongbin; Lee, Jae‐Wook; Kim, Il Kon; Park, Jae‐Han; Park, Chan-Wook; Song, Yosub; Yang, B. S.; Lee, Hyungdeok; Kim, Se‐Ho; Kwon, Daehan; Lee, Seongju; Kim, Kyuyoung; Oh, Sae-Sook; Park, Joonhong; Hong, Gi-Moon; Ka, Dongyoon; Hwang, Kyu-Dong; Park, Jeongje; Kang, Kyeongpil; Kim, Jungyeon; Jeon, Junyeol; Lee, Myeongjun; Shin, Min-Young; Shin, Min-Hwan; Cha, Jaekyung; Jung, Changson; Chang, Kijoon; Jeong, Chunseok; Lim, Euicheol; Park, Il; Chun, Junhyun; Hynix, Sk

doi:10.1109/hcs55958.2022.9895629

Cited by 18 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To support the second and third methods, we should flush cached PIM operands into DRAM before the PIM execution and invalidate them. The cache flush and invalidation incur significant overhead, so most PIM studies declared the PIM operands as uncached attributes [22], [23], [24], [25], [26]. However, the CPU access to the uncached data is too slow because of their strictly ordered memory operations.…”

Section: A Pim Execution Behavior Vs Dmamentioning

confidence: 99%

“…Processing-in-Memory (PIM) architectures have been actively studied by placing computing units close to [9], [10], [11], [12], and [13] or inside memory [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26] to overcome the memory bandwidth limitation. PIM can maximize internal memory bandwidth for the computation using bank-level parallelism [14], [15], [17], [18], [22], [23], [24], [25], [26], thus providing high computation performance. For example, the decoupled PIM [26] achieved a speedup of 75.8x and 1.2x over CPU and GPU at the Level-3 BLAS, respectively.…”

Section: Introductionmentioning

confidence: 99%

“…Second and more importantly, the PIM instruction set architecture (ISA) is unavailable, which fully supports compatibility with commercial computing platforms and perfectly fits the PIM concept and the PIM-target application characteristic. Most PIMs developed their own PIM instructions [11], [12], [22], [23], [30] and offloaded them to the PIM engine by modifying the bus interface and using the CPU load/store instructions. It increases the hardware cost and raises the system performance issues, such as making the core very busy and incurring high latency to access uncacheable PIM areas.…”

Section: Introductionmentioning

confidence: 99%

“…It increases the hardware cost and raises the system performance issues, such as making the core very busy and incurring high latency to access uncacheable PIM areas. Recent PIMs use Direct Memory Access (DMA) as the offloading mechanism [22], [25], [26], [30], [32] to resolve the performance issue by transferring opcodes and large-size operands without CPU intervention.…”

Section: Introductionmentioning

confidence: 99%

“…When processing a transaction, i.e., each PISA-DMA, we can execute memory requests in parallel across banks by exploiting the bank-level parallelism and in-order inside a bank. The separate opcode and operand offloading in the previous works [22], [32] asked a user to carefully program for synchronizing the PIM memory requests between the opcode and the operand execution, thus incurring high programming complexity and related execution overhead.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

PISA-DMA: Processing-in-Memory Instruction Set Architecture Using DMA

et al. 2023

View full text Add to dashboard Cite

Processing-in-memory (PIM) has attracted attention to overcome the memory bandwidth limitation, especially for computing memory-intensive DNN applications. Most PIM approaches use the CPU's memory requests to deliver instructions and operands to the PIM engines, making a core busy and incurring unnecessary data transfer, thus, resulting in significant offloading overhead. DMA can resolve the issue by transferring a high volume of successive data without intervening CPU and polluting the memory hierarchy, thus perfectly fitting the PIM concept. However, the small computing resources of DRAM-based PIM devices allow us to transfer only small amounts of data at one DMA transaction and require a large number of descriptors, thus still incurring significant offloading overhead. This paper introduces PIM Instruction Set Architecture (ISA) using a DMA descriptor called PISA-DMA to express a PIM opcode and operand in a single descriptor. Our ISA makes PIM programming intuitive by thinking of committing one PIM instruction as completing one DMA transaction and representing a sequence of PIM instructions using the DMA descriptor list. Also, PISA-DMA minimizes the offloading overhead while guaranteeing compatibility with commercial platforms. Our PISA-DMA eliminates the opcode offloading overhead and achieves 1.25x, 1.31x, and 1.29x speedup over the baseline PIM at the sequence length of 128 with the BERT, RoBERTa, and GPT-2 models, respectively, in ONNX runtime with real machines. Also, we study how our proposed PISA affects performance in compiler optimization and show that the operator fusion of matrix-matrix multiplication and element-wise addition achieved 1.04x speedup, a similar performance gain using conventional ISAs.

show abstract

Section: A Pim Execution Behavior Vs Dmamentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

PISA-DMA: Processing-in-Memory Instruction Set Architecture Using DMA

et al. 2023

View full text Add to dashboard Cite

show abstract

PIM GPT a hybrid process in memory accelerator for autoregressive transformers

Wu,

Wang,

2024

npj Unconv. Comput.

View full text Add to dashboard Cite

Decoder-only Transformer models such as Generative Pre-trained Transformers (GPT) have demonstrated exceptional performance in text generation by autoregressively predicting the next token. However, the efficiency of running GPT on current hardware systems is bounded by low compute-to-memory-ratio and high memory access. In this work, we propose a Process-in-memory (PIM) GPT accelerator, PIM-GPT, which achieves end-to-end acceleration of GPT inference with high performance and high energy efficiency. PIM-GPT leverages DRAM-based PIM designs for executing multiply-accumulate (MAC) operations directly in the DRAM chips, eliminating the need to move matrix data off-chip. Non-linear functions and data communication are supported by an application specific integrated chip (ASIC). At the software level, mapping schemes are designed to maximize data locality and computation parallelism. Overall, PIM-GPT achieves 41 − 137 × , 631 − 1074 × speedup and 123 − 383 × , 320 − 602 × energy efficiency over GPU and CPU baseline on 8 GPT models with up to 1.4 billion parameters.

show abstract