The Development of Progress Plans Using a Performance‐Based Expert Judgment Model to Assess Technical Performance and Risk

Abstract

Systems engineers are routinely tasked with facilitating the delicate balance between cost, schedule, and technical performance in acquisition programs that are continuously subjected to various outside influences. While there are several quantitative methods to estimate acquisition program cost and schedule performance as well as identify their risks (e.g., Earned Value Management), the estimation of technical performance and technical risk is generally heuristic in nature. In order to monitor the progress of the technical aspects of an acquisition program, the systems engineering discipline utilizes the process of tracking Technical Measures to gain insight into the design and development, to assess risks and issues, and to evaluate the likelihood of realizing objectives. However, with the diversity of so many technical programs, the estimation and risk analysis of technical performance in technology acquisition programs rely on the opinions of experts because the identification and application of relevant quantitative data for constructive modeling is not practical. The Expert-weighted Technical Risk Index methodology proposed in this article introduces a well-established method for mathematically combining expert judgment into the realm of systems engineering to develop predictive progress plans for technical performance estimation and risk analysis.

Improving expert forecasts in reliability: Application and evidence for structured elicitation protocols

“…There is a perception that better performing experts can be selected or weighted on the basis of their credentials . We found little evidence to support this contention (Supporting Information 1: Section ).…”

“…In reliability, judgements used to inform decisions and assessments are often derived by a single expert and may be made heuristically . Our study revealed the serious risk such practices may pose (particularly in terms of overconfidence) and demonstrated that key steps of structured elicitation protocols can improve the final judgements.…”

“…As stated by Bedford et al, typically such judgements are adjusted through multiplication factors; however, the methods employed are often not supported by clear and transparent expert judgement protocols or models. Other researchers have supported this contention indicating that in their experience judgements are often made heuristically or represent the guesstimates of a single individual …”

“…Defence organisations procure equipment to enhance future capabilities. Despite substantial resource investment, poor procurement decisions are commonplace. Such mistakes are costly, and they can leave defence organisations unable to satisfy future capability challenges .…”

“…Despite substantial resource investment, poor procurement decisions are commonplace. Such mistakes are costly, and they can leave defence organisations unable to satisfy future capability challenges . Improved forecasts of failures prior to purchase and operation would improve defence agency decisions.…”