Prodigy: Improving the Memory Latency of Data-Indirect Irregular Workloads Using Hardware-Software Co-Design

Abstract

Irregular workloads are typically bottlenecked by the memory system. These workloads often use sparse data representations, e.g., compressed sparse row/column (CSR/CSC), to conserve space at the cost of complicated, irregular traversals. Such traversals access large volumes of data and offer little locality for caches and conventional prefetchers to exploit. This paper presents Prodigy, a low-cost hardware-software codesign solution for intelligent prefetching to improve the memory latency of several important irregular workloads. Prodigy targets irregular workloads including graph analytics, sparse linear algebra, and fluid mechanics that exhibit two specific types of datadependent memory access patterns. Prodigy adopts a "best of both worlds" approach by using static program information from software, and dynamic run-time information from hardware. The core of the system is the Data Indirection Graph (DIG)-a proposed compact representation used to express program semantics such as the layout and memory access patterns of key data structures. The DIG representation is agnostic to a particular data structure format and is demonstrated to work with several sparse formats including CSR and CSC. Program semantics are automatically captured with a compiler pass, encoded as a DIG, and inserted into the application binary. The DIG is then used to program a low-cost hardware prefetcher to fetch data according to an irregular algorithm's data structure traversal pattern. We equip the prefetcher with a flexible prefetching algorithm that maintains timeliness by dynamically adapting its prefetch distance to an application's execution pace.We evaluate the performance, energy consumption, and transistor cost of Prodigy using a variety of algorithms from the GAP, HPCG, and NAS benchmark suites. We compare the performance of Prodigy against a non-prefetching baseline as well as stateof-the-art prefetchers. We show that by using just 0.8KB of storage, Prodigy outperforms a non-prefetching baseline by 2.6× and saves energy by 1.6×, on average. Prodigy also outperforms modern data prefetchers by 1.5-2.3×.Index Terms-DRAM stalls, irregular workloads, graph processing, hardware-software co-design, programming model, programmer annotations, compiler, and hardware prefetching. Program the prefetcher Program the prefetcher Generate prefetch requests Generate prefetch requestsCompiler analysis Run application Software Hardware Instrumented application binary Application source code Add DIG representation Data Indirection Graph (DIG)

Crescent: Taming Memory Irregularities for Accelerating Deep Point Cloud Analytics

“…Optimizing Irregular Memory Accesses Recent work has made significant strides in domain-agnostic prefetching for irregular applications [10,43,57]. Our split-tree structure can be seen as an application-specific prefetcher and achieves "perfect prefetching" in that 1) off-chip data accesses are overlapped with computation, 2) data needed by the accelerator are readily available on-chip without stalls, and 3) no redundant DRAM accesses are needed.…”

täkō

“…Prior work has studied many optimizations that transform data as it moves through the cache, e.g., to compress [9,36,90,106,107,118,136,146], decrypt [47,65,115], prefetch [6,131,149], change layout [7,23], memoize [8,40,153,154], or serialize/de-serialize [108] data. We motivate täkō by observing how its onMiss callback enables arbitrary data transformations while improving performance, saving energy, and reducing overall work.…”

“…Specialized cache hierarchies. These trends have been widely recognized, and there are many proposals to accelerate data movement, e.g., in machine learning [2,50], graph analytics [92,95,150], data structures [54,58,154], memoization [8,40,153,154], compression [9,36,90,106,107,118,136,146], data layout [7,23,155], prefetching [6,131,149], coherence and synchronization [34,75,151,152], memory management [85,135], and system software [67,108,127]. While highly effective, they share the drawback of requiring custom hardware.…”

“…The first programmable memory hierarchies were explored in the '90s and focused on distributed cache coherence [3,23,56,73,113,123]. More recently, designs have added some programmability to the memory hierarchy for specific purposes: e.g., prefetching [6,131] or compression [146]. By contrast, täkō targets a much wider set of features and optimizations by providing a general-purpose interface and architecture to increase software's visibility and control over data movement.…”

NDMiner

“…Resolving load-dependent control flow operations at NDP precludes the need for using expensive branch resolution mechanisms on the CPU. Moreover, irregular accesses to graph data structures resulting in high memory latency and/or bandwidth [33,54] can be better serviced near memory at a low latency and high available bandwidth, addressing the two main bottlenecks in GPM workloads (Takeaway 3). In summary, NDP is an attractive candidate for accelerating GPM workloads.…”