Memory system performance of UNIX on CC-NUMA multiprocessors

Abstract

This study characterizes the performance of a variant of UNIX SVR4 on a large shared-memory multiprocessor and analyzes the effects of possible OS and architectural changes. We use a nonintrusive cache miss monitor to trace the execution of an OS-intensive multiprogrammed workload on the Stanford DASH, a 32-CPU CC-NUMA multiprocessor (CC-NUMA multiprocessors have cache-coherent shared memory that is physically distributed across the machine). We find that our version of UNIX accounts for 24% of the workload's total execution time. A surprisingly large fraction of OS time (79%) is spent on memory system stalls, divided equally between instruction and data cache miss time. In analyzing techniques to reduce instruction cache miss stall time, we find that replication of only 770 of the OS code would allow 80% of instruction cache misses to be serviced locally on a CC-NUMA machine. For data cache misses, we find that a small number of routines account for 96?70of OS data cache stall time. We find that most of these misses are coherence (communication) misses, and larger caches will not necessarily help. After presenting detailed performance data, we analyze the benefits of several OS changes and predict the effects of altering the cache configuration, degree of clustering, and cache coherence mechanism of the machine. (This paper is available via http://wwwflash. stanford.edu.)

The impact of architectural trends on operating system performance

“…This kind of error is more difficult to detect since it is likely the workload will continue to run correctly. To validate the timings and statistic reporting, we configure SimOS to look like the one-cluster DASH multiprocessor used in a previous operating system characterization study [2] and examine the cache and profile statistics of a parallel compilation workload. Statistics in [2] were obtained with a bus monitor, and are presented in Table 2.1.…”

“…To validate the timings and statistic reporting, we configure SimOS to look like the one-cluster DASH multiprocessor used in a previous operating system characterization study [2] and examine the cache and profile statistics of a parallel compilation workload. Statistics in [2] were obtained with a bus monitor, and are presented in Table 2.1. Although the sources of these statistics are completely different, the system behavior is quite similar.…”

“…One interesting point of comparison between these studies and ours is the methodology used to observe system behavior. Previous studies were based on the analysis of traces either using hardware monitors [2][8] [13] or through software instrumentation [3]. To these traditional methodologies, we add the use of complete machine simulation for operating system characterization.…”

“…For example, operating system studies using the DASH bus monitor [2] [13] observe only level-2 cache misses and hence are blind to performance effects due to the level-1 caches, write buffer, and processor pipeline. Furthermore, because the caches filter memory references seen by the monitor, only a limited set of cache configurations can be examined.…”

Reducing coherence overhead and boosting performance of high-end SMP multiprocessors running a DSS workload

“…The importance of considering operating system activity has been highlighted in previous works [6,5,42]. Process migration, needed to achieve load balancing in such systems, increases the number of cold/conflict misses and also generates useless coherence overhead, known as passive sharing overhead [16].…”