Dynamic hot data stream prefetching for general-purpose programs

Abstract

Prefetching data ahead of use has the potential to tolerate the growing processor-memory performance gap by overlapping long latency memory accesses with useful computation. While sophisticated prefetching techniques have been automated for limited domains, such as scientific codes that access dense arrays in loop nests, a similar level of success has eluded general-purpose programs, especially pointer-chasing codes written in languages such as C and C++.We address this problem by describing, implementing and evaluating a dynamic prefetching scheme. Our technique runs on stock hardware, is completely automatic, and works for generalpurpose programs, including pointer-chasing codes written in weakly-typed languages, such as C and C++. It operates in three phases. First, the profiling phase gathers a temporal data reference profile from a running program with low-overhead. Next, the profiling is turned off and a fast analysis algorithm extracts hot data streams, which are data reference sequences that frequently repeat in the same order, from the temporal profile. Then, the system dynamically injects code at appropriate program points to detect and prefetch these hot data streams. Finally, the process enters the hibernation phase where no profiling or analysis is performed, and the program continues to execute with the added prefetch instructions. At the end of the hibernation phase, the program is deoptimized to remove the inserted checks and prefetch instructions, and control returns to the profiling phase. For long-running programs, this profile, analyze and optimize, hibernate, cycle will repeat multiple times. Our initial results from applying dynamic prefetching are promising, indicating overall execution time improvements of 5-19% for several memory-performance-limited SPECint2000 benchmarks running their largest (ref) inputs.

Overview of Real-Time Java Computing

“…Examples of adaptive optimization systems include HotSpot virtual machine [64] from Sun (now Oracle), Jikes RVM [65] from IBM, and Open Runtime Platform (ORP) [66] from Intel Corporation. To identify hot-spots, researchers have proposed to use online hardware profiling mechanisms such as counters and samplings [67][68][69][70][71], or to use program instrumentation [72][73][74][75][76][77][78], combined instrumentation and sampling [79][80][81], or coupled offline and online profiling [82]. To further improve adaptive optimization, a number of techniques have been developed; for example, recompilation [83], deferred and partial compilation [84][85][86], and dynamic deoptimization [87].…”

A Key-based Adaptive Transactional Memory Executor

“…The adaptive scheme uses data sampling to estimate the distribution of the workload. Other groups have used code-based sampling to identify data streams and strides [4,22]. While most past techniques exploit dynamic locality for sequential execution, we focus on parallel execution.…”

Last-Touch Correlated Data Streaming

“…Another proposed class of prefetchers utilizes address correlation [3,4,10,11,15,20], which promises wider applicability across a diverse spectrum of workloads because they target generalized memory access patterns. Rather than detecting patterns in data layout, these prefetchers correlate data addresses to predict future misses.…”