Revisiting the Complexity of Hardware Cache Coherence and Some Implications

Abstract

Cache coherence is an integral part of shared-memory systems but is also widely considered to be one of the most complex parts of such systems. Much prior work has addressed this complexity and the verification techniques to prove the correctness of hardware coherence. Given the new multicore era with increasing number of cores, there is a renewed debate about whether the complexity of hardware coherence has been tamed or whether it should be abandoned in favor of software coherence. This article revisits the complexity of hardware cache coherence by verifying a publicly available, state-of-the-art implementation of the widely used MESI protocol, using the Murϕ model checking tool. To our surprise, we found six bugs in this protocol, most of which were hard to analyze and took several days to fix. To compare the complexity, we also verified the recently proposed DeNovo protocol, which exploits disciplined software programming models. We found three relatively easy to fix bugs in this less mature protocol. After fixing these bugs, our verification experiments showed that, compared to DeNovo, MESI had 15X more reachable states leading to a 20X increase in verification (model checking) time. Although we were eventually successful in verifying the protocols, the tool required making several simplifying assumptions (e.g., two cores, one address). Our results have several implications: (1) they indicate that hardware coherence protocols remain complex;(2) they reinforce the need for protocol designers to embrace formal verification tools to demonstrate correctness of new protocols and extensions; (3) they reinforce the need for formal verification tools that are both scalable and usable by non-expert; and (4) they show that a system based on hardware-software co-design can offer a simpler approach for cache coherence, thus reducing the overall verification effort and allowing verification of more detailed models and protocol extensions that are otherwise limited by computing resources.

McVerSi: A test generation framework for fast memory consistency verification in simulation

“…MESI+PUTX-Race*: This bug is caused by a protocol race condition and subsequent invalid transition. It is described in detail by Komuravelli et al [12], who previously discovered it via model checking with Murϕ. This bug does not manifest as a MCM bug directly, but instead is caught by Ruby as an invalid transition.…”

“…a proof of correctness, the model being verified against its specification is typically a component abstraction of what is present in the functional design implementation; with the coherence protocol being the main artifact being subjected to formal verification [14]. For most model checking approaches [7,9,10,12,13], the consistency properties intended to capture MCM correctness are derived properties, such as the SWMR [15] invariant; these are inadequate for protocols explicitly violating such properties (e.g. lazy self-invalidation based protocols using "tear-off" blocks [43]).…”

“…Indeed, the recent model checking of the MESI coherence protocol of the GEMS memory simulator (and of Gem5) has found bugs [12] (MESI+PUTX-Race among others). Independent of the memory system, PipeCheck [16] can be used for model checking of pipeline abstractions (albeit against selected litmus tests), which also uncovered a bug in Gem5 (LQ+no-TSO).…”

“…In the pre-silicon design phase, approaches based on formal methods are usually applied to abstract models of components of a design. For example, model checking of coherence protocols has been studied extensively [7,8,9,10,11,12,13,14]; consistency properties verified are commonly based on derived properties, such as the Single-Writer-Multiple-Reader (SWMR) [15] invariant. Another example is the recently developed PipeCheck [16] tool, which is a domain-specific MCM model checker for pipeline abstractions.…”

DeNovoSync

“…Specifically, the DeNovo project [10,21,35,36] showed that hardware coherence can be simpler and more efficient if shared-memory programs are more "disciplined;" i.e., if parallelism is explicitly requested through structured parallel constructs with memory side-effects statically or dynamically identified. DeNovo eliminates directory storage for sharer lists, writer-initiated invalidation traffic, false sharing, and protocol transient states.…”