Scale-out Systolic Arrays

Abstract

Multi-pod systolic arrays are emerging as the architecture of choice in DNN inference accelerators. Despite their potential, designing multi-pod systolic arrays to maximize effective throughput/Watt—i.e., throughput/Watt adjusted when accounting for array utilization—poses a unique set of challenges. In this work, we study three key pillars in multi-pod systolic array designs, namely array granularity, interconnect, and tiling. We identify optimal array granularity across workloads and show that state-of-the-art commercial accelerators use suboptimal array sizes for single-tenancy workloads. We, then evaluate the bandwidth/latency trade-offs in interconnects and show that Butterfly networks offer a scalable topology for accelerators with a large number of pods. Finally, we introduce a novel data tiling scheme with custom partition size to maximize utilization in optimally sized pods. We propose Scale-out Systolic Arrays , a multi-pod inference accelerator for both single- and multi-tenancy based on these three pillars. We show that SOSA exhibits scaling of up to 600 TeraOps/s in effective throughput for state-of-the-art DNN inference workloads, and outperforms state-of-the-art multi-pod accelerators by a factor of 1.5 ×. 1

SAVector: Vectored Systolic Arrays

“…The size of the global buffer is smaller than the combined size of the SRAM buffers dedicated to each pod. SAVector reduces the overall SRAM overhead by 75% compared to prior work [3]. Furthermore, we observe that our design is suitable for scale-out architectures with hundreds of pods, making SAVector a promising solution for achieving DNN accelerator scalability.…”

“…In our evaluation, SAVector achieves a 52% energy-delayproduct (EDP) reduction compared to the scale-up architecture. Also, SAVector achieves 52% speedup and 27% EDP reduction compared to the prior work [3].…”

“…Prior work has proposed DNN accelerators based on a design approach: a single pod offers high peak throughput, and a single accelerator equips a few (up to four) pods. Prior work refers to this architecture as a "scale-up architecture [3]." For example, Google TPUv4i consists of four pods, each featuring a 128×128 systolic array [4].…”

“…As a solution to the aforementioned challenge, prior work has proposed a scale-out architecture for DNN accelerators [3], [6]- [9]. Unlike the scale-up architecture, the scale-out architecture comprises tens or even hundreds of pods, with each pod having small systolic arrays (e.g., 8×8 or 32×32).…”

“…However, such an on-chip memory structure is not scalable, as an accelerator with tens or hundreds of pods would require hundreds of megabytes of SRAM resources for the dedicated buffers. Prior work has also employed a shared SRAM buffer connected with all pods [3]. Although the shared buffer effectively filters out the duplicated off-chip memory access, the prior work still suffers from performance degradation due to long buffer access latency.…”