11.4.1 Generating Predictive Models Using Decision Trees and Neural Networks for Large‐Scale Systems Engineering

Abstract

Systems engineering must tackle the challenges of computational systems that are increasingly large-scale and software-intensive in terms of system size, component breadth and maturity, and development heterogeneity. This research describes and empirically evaluates techniques for generating predictive models for enabling large-scale system development and management. We describe two types of metric-driven decision models, decision trees and neural networks, which classify software components in large systems according to their likelihood of having user-specified properties such as high fault-proneness or high development effort. The metric-driven decision models enable coarse-grain analysis of large-scale multi-component heterogeneous systems, and they identify high-payoff areas for directing the application of finegrain analysis techniques for fault detection or redesign. The decision models serve as metric integration mechanisms that enable the synergistic use of numerous metrics simultaneously and integrate measurements collected by development tools or infrastructure. Model generation techniques automatically generate the decision models to calibrate them to new projects and organizations.We evaluate the predictive effectiveness of the decision models in terms of correctness, consistency, and completeness using fault and effort data from large NASA systems. Correctness is defined as the percent of components correctly identified, consistency is defined as 100% minus the percent of false positives, and completeness is defined as 100% minus the percent of false negatives. On average, the decision models had 83.44% correctness, 71.96% consistency, and 65.25% completeness in predictions of high fault and high effort software components. The network models had 89.63% correctness, 79.49% consistency, and 69.09% completeness, while the tree models had 77.25% correctness, 64.42% consistency, and 61.40% completeness. Non-parametric ANOVA comparisons showed that the network models were statistically more accurate than the tree models (α < 0.0001).

Adaptive Activity Driven Multi‐Level Hierarchical Prediction of Complex Systems Through Profiling and Feedback

“…They have been used for various purposes. For examples, Barr [Barr 2008] have used them for organization, learning and cooperation, Bode [Bode 1998 ] have used them for decision support in the management of research and development, Breitler [Breitler 2004] for predicting reliability of complex systems, On Cheung [On Cheung 2006] have used them for predicting project performance, Selby [Selby 2006] for prediction in large scale systems engineering while William Chien [Chien 1999] have used them for strategic planning. Finally, they have been intensively used in embedded systems (example [Chtourou 2006]).…”