A methodology for modeling VVT risks and costs

Abstract

The cost of large systems' Verification, Validation, and Testing (VVT) is in the neighborhood of 40% of the total life cycle cost. The cost associated with systems' failures is even more dramatic, often exceeding 10% of industrial organizations turnover. There is a great potential benefit in streamlining and optimizing the VVT process. The first step in accomplishing this aim is to define a VVT strategy and then to quantify the cost and risk associated with carrying it out. This paper provides an overview of the methodologies for risk and cost monitoring for VVT and proposes a novel approach for modeling VVT strategies as decision problems. A quantitative VVT process and risk model is proposed. Due to the nondeterministic nature of risk, simulation is used to generate distributions of possible costs, schedules, and risk outcomes. These distributions represent a probabilistic approach and are analyzed in relation to impact events. The model provides means to explore different VVT strategies for optimizing relevant decision parameters. To demonstrate the proposed procedure the paper describes a case study depicting a planned avionics suite upgrade program for a fighter aircraft. Some simplified partial quantitative results are also presented. © 2003 Wiley Periodicals, Inc. Syst Eng 6: 135–151, 2003

Testing in the incremental design and development of complex products

“…As such testing is closely related to quality (Engel and Barad 2003). In particular, quality may be viewed as the ability of a product to meet explicit or implicit needs, and testing can help to identify and eliminate quality failures.…”

“…Shabi et al (2017) extend the approach with a more comprehensive knowledge elicitation procedure, and apply it to a case study in the aerospace industry with promising feedback. Engel and Barad (2003) also formulate VVT planning as a decision problem, placing greater emphasis on eliciting the risks that VVT is intended to avoid, and arguing that the ideal VVT strategy is strongly dependent on the project context (for example, the number of units produced). Shankar et al (2016) develop a test planning method focusing on the need to systematically generate and prioritize test activities following design change in a product that can be assembled in different configurations.…”

“…Another exception is Huizinga et al (2002), who focus fairly narrowly on virtual fatigue testing in one automotive company. A third exception is Engel and Barad (2003) who provide a quantitative study of VVT in an avionics upgrade project. Their study focuses on eliciting the cost of quality and quantitatively assessing a pilot implementation of the SysTest approach discussed earlier.…”

“…We contend that the topic deserves further research, because it has been suggested that testing is an important component of product development/systems engineering that can consume substantial proportions of the overall effort and cost (Thomke and Bell 2001;Engel and Barad 2003). Furthermore, several researchers agree that there seems to be opportunity for improvement.…”

“…In some respects, software testing is different from hardware testing, for example, because many tests can be automated and run rapidly at low cost in the context of software, which is often not possible when testing hardware (Shabi and Reich 2012). Despite the differences, some of the insights are applicable to both domains (Engel and Barad 2003), since some objectives of testing are broadly similar in both cases, and it can also be noted that most complex engineered products comprise a significant proportion of software. Although software testing is not the main focus of this paper, the literature on this topic is, therefore, mentioned where appropriate to support the discussion.…”