Optimal bypass monitor for high performance last-level caches

Abstract

In the last-level cache, large amounts of blocks have reuse distances greater than the available cache capacity. Cache performance and efficiency can be improved if some subset of these distant reuse blocks can reside in the cache longer. The bypass technique is an effective and attractive solution that prevents the insertion of harmful blocks.Our analysis shows that bypass can contribute significant performance improvement, and the optimal bypass can achieve similar performance compared to OPT+B, which is the theoretical optimal replacement policy. Thus, we propose a bypass technique called Optimal Bypass Monitor (OBM), which makes bypass decisions by learning and predicting the behavior of the optimal bypass. OBM keeps a short global track of the incoming-victim block pairs. By detecting the first reuse block in each pair, the behavior of the optimal bypass on the track can be asserted to guide the bypass choice.Any existing replacement policy can be extended with OBM while requiring negligible design modification. Our experimental results show that using less than 1.5KB extra memory, OBM with the NRU replacement policy outperforms LRU by 9.7% and 8.9% for single-thread and multiprogrammed workloads respectively. Compared with other state-of-the-art proposals such as DRRIP and SDBP, it achieves superior performance with less storage overhead.

Applying SVM to data bypass prediction in multi core last-level caches

“…A number of works focus on predicting which data will be reused using predictors and information collected based on either the program counter (PC) [2,4,5] or the address [6,7,8].…”

“…The memory system, especially shared last-level cache (LLC), is impacted by these accesses; different types of concurrent workloads result in lower hit rate or even thrashing when the working set is larger than the cache size. In order to better manage limited cache space, selectively bypassing data from the LLC has been shown to ameliorate the problem [1,2,3].…”

“…Temporal locality in the LLC is inverted from inner-level caches [2]; several LLC blocks are never reused before eviction, deteriorating its performance. Recent works proposed methods to improve LLC hit-rate by avoiding allocating every datum, selectively bypassing not-reused data.…”

Adaptive Cache Bypassing for Inclusive Last Level Caches

“…On the other hand, many industrial designs, including Intel Core i7 [26], use inclusive last level caches which makes employing the cache bypassing on these designs a non-trivial task. As suggested in a recent work on cache bypassing algorithm [18], a bypassing algorithm can be modified to work with inclusive caches by inserting the bypassed block at the LRU (least recently used) position. In this manner, the bypassing candidates chosen by the bypassing algorithm are victimized on the next miss to the cache set.…”

“…Recently proposed replacement algorithms [3] [6] can be an effective method to improve the LLC performance for a wide variety of workloads. Recent work by Li et al [18] uses a similar mechanism to decide the effectiveness of bypassing. The key difference is they do not use probabilistic bypassing and instead maintain a signature based history per program counter to make bypassing decisions.…”