Workload Behavior Driven Memory Subsystem Design for Hyperscale

Abstract

Hyperscalars run services across a large fleet of servers, serving billions of users worldwide. However, these services exhibit different behaviors compared to commonly available benchmark suites, leading to server architectures that are suboptimal for cloud workloads. As datacenters emerge as the primary server processor market, optimizing server processors for cloud workloads by better understanding their behavior is an area of interest. To address this, we present MemProf, a memory profiler that profiles the three major reasons for stalls in cloud workloads: code-fetch, memory bandwidth, and memory latency. We use MemProf to understand the behavior of cloud workloads at Meta and propose and evaluate micro-architectural and memory system design improvements that help cloud workloads' performance.MemProf's code analysis shows that cloud workloads at Meta execute the same code across CPU cores. Using this, we propose shared micro-architectural structures-a shared L2 I-TLB and a shared L2 cache. Next, to help with memory bandwidth stalls, using workloads' memory bandwidth distribution, we find that only a few pages contribute to most of the system bandwidth. We use this finding to evaluate a new high-bandwidth, small-capacity memory tier and show that it performs 1.46× better than the current baseline configuration. Finally, we look into ways to improve memory latency for cloud workloads. Profiling using MemProf reveals that L2 hardware prefetchers, which are commonly used to reduce memory latency, have very low coverage and consume a significant amount of memory bandwidth. To help improve future hardware prefetcher performance, we built an efficient memory tracing tool to collect and validate production memory access traces. Our memory tracing tool adds significantly less overhead than DynamoRIO, enabling tracing production workloads.