On the Selection of Error Model(s) for OS Robustness Evaluation

Abstract

The choice of error model used for robustness evaluation of Operating Systems (OSs) influences the evaluation run time, implementation complexity, as well as the evaluation precision. In order to find an "effective" error model for OS evaluation, this paper systematically compares the relative effectiveness of three prominent error models, namely bit-flips, data type errors and fuzzing errors using fault injection at the interface between device drivers OS. Bit-flips come with higher costs (time) than the other models, but allow for more detailed results. Fuzzing is cheaper to implement but is found to be less precise. A composite error model is presented where the low cost of fuzzing is combined with the higher level of details of bit-flips, resulting in high precision with moderate setup and execution costs.

The impact of fault models on software robustness evaluations

“…We reuse the SWIFI framework of [15,18] and extend it to perform comparisons, as initiated in [24]. In the following section we extend the work presented in [18] and develop generalized and formally accurate definitions the informally specified metrics so that they can be used for (a) any robustness evaluation that aligns with the generic method described in Section 2.1 and (b) quantitative comparisons, i.e.…”

“…In the following section we extend the work presented in [18] and develop generalized and formally accurate definitions the informally specified metrics so that they can be used for (a) any robustness evaluation that aligns with the generic method described in Section 2.1 and (b) quantitative comparisons, i.e. we give quantitative re-definitions providing ratio scale measures for all considered metrics.…”

“…Johansson et al [18] compared the effects of three different fault models for interface injections on robustness evaluations of Windows CE .NET. However, the applied metrics were only suited for this specific evaluation and restricted to a specific CUE failure mode, i.e.…”

“…We adopt the fault model notion from [16,18,17], where fault models 3 for SWIFI-based robustness evaluations of operating systems (OSs) were defined by three basic attributes: the fault location (where to inject), the fault type (what to inject), and the fault timing (when to inject), where the latter was further qualified as injection trigger and fault latency. Table 1 lists a number of existing SWIFI frameworks that have been applied for software robustness evaluations similar to those we consider in this paper, along with their reported fault model support.…”

“…Software-implemented fault injection (SWIFI) is a widely used approach for evaluating the robustness of software components. Recent research [24,18] indicates that the selection of the applied fault model has considerable influence on the results of SWIFI-based evaluations, thereby raising the question how to select appropriate fault models (i.e. that provide justified robustness evidence).…”