Pursuing the performance potential of dynamic cache line sizes

Abstract

In this paper we examine the application of offline algorithms for determining the optimal sequence of loads and superloads (a load of multiple consecutive cache lines) for direct-mapped caches. We evaluate potential gains in terms of miss rate and bandwidth and find that in many cases optimal superloading can noticeably reduce the miss rate without appreciably increasing bandwidth. Then we examine how this performance potential might be realized. We examine the effectiveness of a dynamic online algorithm and of static analysis (profiling) for superloading and compare these to next-line prefetching. Experimental results show improvements comparable to those of the optimal algorithm in terms of miss rates. §1 IntroductionSince their introduction over thirty years ago, caches have become ubiquitous as components of the memory hierarchy. Caches have been successful because programs exhibit locality: spatial locality, the tendency for neighboring memory locations to be referenced close together in time; and temporal locality, the tendency for referencing in the future those locations that have been referenced in the recent past. However, as the speed of processors increases much faster than the decrease in memory latency, the efficiency of caches has received more scrutiny.Combinations of hardware and software techniques have been proposed and often implemented to improve locality and to reduce or tolerate memory latency. The basic goal is to reduce cache miss rates without unduly increasing the amount of bytes transferred between levels of the memory hierarchy. When couched in terms of improving spatial locality for data caches, the main theme of this paper, the usual policy is to support larger cache lines. Potential detrimental effects of this policy are a possible increase in cache miss rate because of more frequent conflict misses and the lack of reuse of portions of the larger lines, and to lengthen the occupancy of the bus between levels of the memory hierarchy servicing the miss. In order to palliate these effects and to take advantage of large lines, when deemed profitable, we examine the potential benefit of implementing the cache controller so that on a miss, either the missing regular size line is loaded -hereafter called the base case -or the line is superloaded, i.e., the missing line and surrounding lines are brought into the cache. Note that the advantages of superloading depend on the cost model for the level of the memory hierarchy under investigation. Of particular importance are the relative costs of a load and a superload.Although the impact of these techniques has been investigated using heuristics and software or hardware assists, how much can be gained if these techniques were used optimally is not known. In this paper, after briefly introducing an optimal offline algorithm for choosing between loads and superloads, we derive the margin of maximum possible improvement on the integer Spec95 benchmark suite. We then analyze the performances, when compared to the optimal and base cases, ...

Accurate and Complexity-Effective Spatial Pattern Prediction

“…Vleet et al [24] proposed using off-line profiling to determine the fetch size upon a cache miss. However, the lack of dynamism render these static approaches less effective when faced with a data set that changes rapidly during program execution.…”

“…Recent research [13,17,18,23,24] indicates that there are large spatial variations in cache line usage both within and across programs. In the presence of such drastic variations, a fixed cache line size results in a sub-optimal design point.…”

Reducing DRAM latencies with an integrated memory hierarchy design

“…This expectation may be based on profile information [9,25], hardware detection of strided accesses [17] or spatial locality [ 12,14,25], or compiler annotation of load instructions [23]. Optimal off-line algorithms for fetching a set of noncontiguous words [24] or a variable-sized aligned block [25] on each miss provide bounds on these techniques. Pollution may also be reduced by prefetching into separate buffers [13,23].…”

“…Przybylski [ 181 analyzed cancelling an ongoing demand fetch (after the critical word had returned) on a subsequent miss, but found that performance was reduced, probably because the original block was not written into the cache. Our scheduling technique is independent of the scheme used to generate prefetch addresses; determining the combined benefit of scheduling and more conservative prefetching techniques [9,12,14,17,25] is an area of future research. Our results also show that in a large secondary cache, controlling the replacement priority of prefetched data appears sufficient to limit the displacement of useful referenced data.…”

Spatial Memory Streaming

“…Vleet and co-authors [28] propose offline profiling to select fetch size upon a miss. Guided region prefetching [29] uses compiler hints to direct both spatial and non-spatial (e.g., pointer-chasing) prefetches.…”