High performing cache hierarchies for server workloads: Relaxing inclusion to capture the latency benefits of exclusive caches

Abstract

Increasing transistor density enables adding more on-die cache real-estate. However, devoting more space to the shared lastlevel-cache (LLC) causes the memory latency bottleneck to move from memory access latency to shared cache access latency. As such, applications whose working set is larger than the smaller caches spend a large fraction of their execution time on shared cache access latency. To address this problem, this paper investigates increasing the size of smaller private caches in the hierarchy as opposed to increasing the shared LLC. Doing so improves average cache access latency for workloads whose working set fits into the larger private cache while retaining the benefits of a shared LLC. The consequence of increasing the size of private caches is to relax inclusion and build exclusive hierarchies. Thus, for the same total caching capacity, an exclusive cache hierarchy provides better cache access latency.We observe that server workloads benefit tremendously from an exclusive hierarchy with large private caches. This is primarily because large private caches accommodate the large code workingsets of server workloads. For a 16-core CMP, an exclusive cache hierarchy improves server workload performance by 5-12% as compared to an equal capacity inclusive cache hierarchy. The paper also presents directions for further research to maximize performance of exclusive cache hierarchies.

The Architectural Implications of Facebook's DNN-Based Personalized Recommendation

“…On the other hand, the Skylake architecture has support for AVX-512 instructions, more parallel cores, and larger L2 caches. Furthermore, Haswell and Broadwell implement an inclusive L2/L3 cache hierarchy, while Skylake implements an non-inclusive/exclusive cachehierarchy [34,35]. (For the remainder of this paper we will refer to Skylake's L2/L3 cache hierarchy as exclusive).…”

Toward multi-programmed workloads with different memory footprints: a self-adaptive last level cache scheduling scheme

“…Xiao et al [24] presented a dual queues cache replacement algorithm based on sequentiality detection to improve the cache design. Jaleel et al [23] presented directions for further research to maximize performance of exclusive cache hierarchies. Chou et al [35] proposed CAMEO which not only makes stacked DRAM visible as part of the memory address space but also exploits data locality.…”

“…Most of recent researches on hybrid SRAM and DRAM caches focus mainly on enhancing the overall performance of SRAM (resp., DRAM) by utilizing the merit of DRAM (resp., SRAM). There are also many papers devoted to investigating workload performance: (1) For multi-programmed workloads, prior work discussed the issues of relieving memory contention [10,11], workload balance [12,13] and power related optimization [14]; (2) To improve the performance of memory-intensive workloads, many solutions (e.g., architecture design [15][16][17], OS level method [18][19][20] and feedback control [21,22]) have also been proposed; (3) In the cache system, the improved cache architectures [4,9,23,24] and 3D-stacked DRAM technologies [25][26][27] are used to achieve better workload performance; and so on (a broader overview of related work will be covered in Section 2). Instead, little attention has been paid to designing a last level cache (LLC) scheduling scheme for multi-programmed workloads with different memory footprints.…”

Linux apps-usage-driven power dissipation-aware scheduler

“…Our work focuses on mitigating the power dissipation caused by the following cache coherence problems: a) Non sequential data fetch: Cache prefetcher fetches data in a sequential manner, 978-1-4799-5341-7/16/$31.00 ©2016 IEEE randomly allocated data causes more cache misses. One way to ensure sequential data fetching is to redesign the cache hierarchy [10]. However, it is difficult to keep data allocation in sequence in SMT CMP architecture, where the context switches regularly and memory gets allocated randomly.…”