Emulation of Transient Software Faults for Dependability Assessment: A Case Study

Abstract

Fault Tolerance Mechanisms (FTMs) are extensively used in software systems to counteract software faults, in particular against faults that manifest transiently, namely Mandelbugs. In this scenario, Software Fault Injection (SFI) plays a key role for the verification and the improvement of FTMs. However, no previous work investigated whether SFI techniques are able to emulate Mandelbugs adequately. This is an important concern for assessing critical systems, since Mandelbugs are a major cause of failures, and FTMs are specifically tailored for this class of software faults.In this paper, we analyze an existing state-of-the-art SFI technique, namely G-SWFIT, in the context of a real-world fault-tolerant system for Air Traffic Control (ATC). The analysis highlights limitations of G-SWFIT regarding its ability to emulate the transient nature of Mandelbugs, because most of injected faults are activated in the early phase of execution, and they deterministically affect process replicas in the system. We also notice that G-SWFIT leaves untested the 35% of states of the considered system. Moreover, by means of an experiment, we show how emulation of Mandelbugs is useful to improve SFI. In particular, we emulate concurrency faults, which are a critical sub-class of Mandelbugs, in a fully representative way. We show that proper fault triggering can increase the confidence in FTMs' testing, since it is possible to reduce the amount of untested states down to 5%.

MoIO: Run-time monitoring for I/O protocol violations in storage device drivers

“…This idea is based on the observation that device driver failures have a transient nature, and are triggered by relatively-rare environmental factors that are difficult to reproduce and to debug [21], [10], [12], [22], [23], [24], [25]. Such environmental factors include multi-threading, asynchrony, interrupts, locking protocols, hardware events and virtual memory [21].…”

Towards identifying OS-level anomalies to detect application software failures

“…In order to accelerate the collection of failure related data, namely statistics of the behavior of the system when some faults are activated, we perform some tests by injecting bugs into the SWIMBOX source code (for each run we inject just a bug). We achieve this by using a tool developed for Fault Injection Test [11], which adopts source code mutation operators [12] designed to emulate most commonly developer errors.…”

Timing Predictability in Future Multi-Core Avionics Systems

“…In this thesis, we refer to these kinds of bugs as transient software faults. Other terms for software faults are Bohrbug (permanent), Mandelbug and Heisenbug (transient) [29,67]. The majority of software faults found in production software are transient as most of the other bugs, those that always fail, have hopefully been discovered in the process of design, review and testing [29].…”

“…Natella and Cotroneo [67] investigate whether SFI really does emulate mandelbugs (transient software faults) to a satisfactory degree. They perform a case study and analyze the SFI tool G-SWFIT, finding that the injected faults do not represent mandelbugs that well.…”

“…Software faults that do not show in every execution, sometimes referred to as mandelbugs, where the effect is transient in nature are not well represented by existing SFI tools [67]. This type of bugs is hard to find during design time and pre-production testing.…”