Experimental Comparison of Two Safety Analysis Methods and Its Replication

Abstract

Background) Empirical Software Engineering (SE) strives to provide empirical evidence about the pros and cons of SE approaches. This kind of knowledge becomes relevant when the issue is whether to change from a currently employed approach to a new one or not. An informed decision is required and is particularly important in the development of safety-critical systems. For example, for the safety analysis of safety-critical embedded systems, methods such as Failure Mode and Effect Analysis (FMEA) and Fault Tree Analysis (FTA) are used. With the advent of model-based systems and software development, the question arises whether safety engineering methods should also be adopted. New technologies such as Component Integrated Fault Trees (CFT) come into play. Industry demands to know the benefits of these new methods over established ones such as Fault Trees (FT). (Methods) For the purpose of comparing CFT and FT with regard to the capabilities of the safety analysis methods (such as quality of the results) and to the participants' rating of the consistency, clarity, and maintainability of the methods, we designed a comparative study as a controlled experiment using a within-subject design. The experiment was run with seven academic staff members working towards their PhD. The study was replicated with eleven domain experts from industry. (Results) Although the analysis of the tasks' solutions showed that the use of CFT did not yield a significantly different number of correct or incorrect solutions, the participants rated the modeling capacities of CFT higher in terms of model consistency, clarity, and maintainability. (Conclusion) From this first evidence, we conclude that CFT have the potential of being beneficial for companies looking for a safety analysis approach for projects using model-based development.

Publication Bias

“…≠ The previous elements (research goal/question/hypothesis, and statistical hypothesis) do not usually appear together in the same experiment. Jung et al [13] is one of the few cases in which the four are explicitly reported. In turn, it is rather common that one or several are missing (in particular, the research hypotheses and the statistical hypotheses).…”

“…Although unnecessary from a technical viewpoint, researchers tend to create research questions to report this refinement process explicitly. For instance, the research questions below appear also in [13]:…”

“…We wanted to find out whether CFT are less error-prone and whether the participants would favor CFT over FT [13]. • Research hypothesis: A declaration of the relationship between independent and dependent variables that drives the research design [19].…”

“…• Research hypothesis: A declaration of the relationship between independent and dependent variables that drives the research design [19]. For instance: 21 : When using CFT, consistency between the system description and the safety analysis model is perceived differently than when using FT. [13] • Research questions: A research goal breaks down into several research hypothesis rather frequently. Although unnecessary from a technical viewpoint, researchers tend to create research questions to report this refinement process explicitly.…”

“…Although unnecessary from a technical viewpoint, researchers tend to create research questions to report this refinement process explicitly. For instance, the research questions below appear also in [13]: 1 : Will the application of the CFT yield the same quality of the resulting safety model as a model built with FT?…”