The Quantitative Estimation of IT‐Related Risk Probabilities

Abstract

How well can people estimate IT-related risk? Although estimating risk is a fundamental activity in software management and risk is the basis for many decisions, little is known about how well IT-related risk can be estimated at all. Therefore, we executed a risk estimation experiment with 36 participants. They estimated the probabilities of IT-related risks and we investigated the effect of the following factors on the quality of the risk estimation: the estimator's age, work experience in computing, (self-reported) safety awareness and previous experience with this risk, the absolute value of the risk's probability, and the effect of knowing the estimates of the other participants (see: Delphi method). Our main findings are: risk probabilities are difficult to estimate. Younger and inexperienced estimators were not significantly worse than older and more experienced estimators, but the older and more experienced subjects better used the knowledge gained by knowing the other estimators' results. Persons with higher safety awareness tend to overestimate risk probabilities, but can better estimate ordinal ranks of risk probabilities. Previous own experience with a risk leads to an overestimation of its probability (unlike in other fields like medicine or disasters, where experience with a disease leads to more realistic probability estimates and nonexperience to an underestimation).

Risk assessment of the Istanbul Strait by using Ports and Waterways Safety Assessment (PAWSA) method

“…Another application discusses effectiveness of community-based environmental protection policy by using a Delphi-fuzzy method [15]. Herrmann considers IT-related risk probabilities [16]. Moreover, another Delphi application is conducted by Karin et al in a step-wise manner [17].…”

Analysis of Alternatives for Risk Assessment Methodologies and Tools

“…A review paper of the Delphi method suggests "there is no consistent evidence that the technique outperforms other structured group procedures," but the authors acknowledge the "sheer variety of technique formats that have been used as representative of Delphi" makes the assessment difficult [31]. Discussion in [32] summarizes how the design of elicitation techniques for the Delphi method and other closely related techniques (such as nominal group theory) can impact the results.…”

FMEA-AI: AI fairness impact assessment using failure mode and effects analysis

“…We do not explicitly list detection or inheritance in our FMEA-AI sample worksheets as this work explored only the basic concepts of FMEA to demonstrate its suitability for impact assessments. A second issue with FMEA is that calculating severity and probability ratings typically requires data [66]. Although FMEA-AI ratings may be estimated in a more qualitative way as is done with other impact assessments, more accurate ratings of severity and probability may be achieved through incorporating historical trend data, pilot test data, field data or simulation data.…”