Defining Better Test Strategies with Tradespace Exploration Techniques and Pareto Fronts: Application in an Industrial Project

Abstract

Test strategies are usually defined following a point-design like method. Starting with a set of verification requirements and programmatic constraints, and usually with a generic test sequence, a baseline test approach is defined. Then, the baseline is optimized until an acceptable strategy is found. Academia has consistently shown however the benefits of using tradespace exploration techniques instead of pointbased designs. Some industrial applications seem to corroborate such findings. Yet, both academia and industry have limited the use of tradespace exploration techniques to selecting design concepts. This paper proposes that broadening the use of tradespace exploration techniques and Pareto frontiers to other activities in the field of systems engineering domain yields similar benefits. In particular, this paper presents the actual application of tradespace exploration techniques and Pareto frontiers in an industrial context to select a test strategy for a system. This paper provides thus three main contributions. First, the paper demonstrates that tradespace exploration and Pareto frontiers can be beneficial beyond concept selection. Second, the paper presents a process to use tradespace exploration techniques and Pareto frontiers for selecting test strategies. Finally, the paper showcases the application of the proposed technique and process in a real industrial setup, which yielded a number of lessons learnt. C⃝ 2016 Wiley Periodicals, Inc. Syst Eng 18: 639-658, 2015

A Concept for Set‐based Design of Verification Strategies

“…Consider on the contrary, that the verification activity 1 V at 3 t showed much better results than previously expected. This may yield a higher than expected confidence on the system being absent of errors, potentially making verification activity 2 V at 3 t unnecessary or of little value, because of how confidence builds up on prior information (Salado andKannan, 2018b, Salado et al, 2018).…”

“…Under a common master plan, low level verification activities are executed as risk mitigation activities, such as early identification of problems, or because some of them are not possible at higher levels of integration (Engel, 2010). Therefore, a verification strategy is defined "aiming at maximizing confidence on verification coverage, which facilitates convincing a customer that contractual obligations have been met; minimizing risk of undetected problems, which is important for a manufacturer's reputation and to ensure customer satisfaction once the system is operational; and minimizing invested effort, which is related to manufacturer's profit" (Salado, 2015). Essentially, verification activities are the vehicle by which contractors can collect evidence of contractual fulfillment in acquisition programs.…”

“…Hence, the resources necessary to execute verification activities at various stages of the system development are allocated and committed at the beginning, when a small amount of knowledge about the system is available (Engel, 2010). However, the necessity and value of a verification activity cannot be measured independently of the overall verification strategy (Salado andKannan, 2018b, Salado et al, 2018). Instead, the necessity to perform a given verification activity depends on the results of all verification activities that have been previously performed.…”

Elemental patterns of verification strategies

“…“Prescriptive models” apply mathematics and optimization to designing verification strategies (eg, ). However, the mathematical model is limited to reflect only certain aspects of verification strategies.…”

A Combinatorial Approach to Tradespace Exploration of Complex Systems: A CubeSat Case Study

“…Different from optimization, exploration allows system engineers to explore a larger design space without converging onto a solution too early and missing viable system alternative opportunities (MacCalman, Beery et al 2016). Tradespace exploration recognize that a set of optimal solutions exists as opposed to single optimum solutions in multi-attribute decisions (Salado 2015).…”