Classification Accuracy of Actuarial Risk Assessment Instruments

Abstract

Users of commonly employed actuarial risk assessment instruments (ARAIs) hope to generate numerical probability statements about risk; however, ARAI manuals often do not explicitly report data that are essential for understanding the classification accuracy of the instruments. In addition, ARAI manuals often contain data that have the potential for misinterpretation. The authors of the present article address the accurate generation of probability statements. First, they illustrate how the reporting of numerical probability statements based on proportions rather than predictive values can mislead users of ARAIs. Next, they report essential test characteristics that, to date, have gone largely unreported in ARAI manuals. Then they discuss a graphing method that can enhance the practice of clinicians who communicate risk via numerical probability statements. After the authors review several strategies for selecting optimal cut-off scores, they show how the graphing method can be used to estimate positive predictive values for each cut-off score of commonly used ARAIs, across all possible base rates. They also show how the graphing method can be used to estimate base rates of violent recidivism in local samples.

The Effect of Sample Heterogeneity and Risk Categorization on Area Under the Curve Predictive Validity Metrics

“…In the alternative binormal ROC method, the AUC can be derived from the mean and standard deviation of risk scores for recidivists and nonrecidivists, but this AUC will be valid if the score distributions of both groups follow normal distributions, or could do so if monotonically transformed. The validation samples of several instruments in widespread use do not follow normal distributions (Neller & Frederick, 2013). It is not clear whether they could be transformed successfully, and I suggest that using the trapezoid method is a safer alternative that avoids the risk of violating the parametric assumption.…”

Predictive Validity Performance Indicators in Violence Risk Assessment: A Methodological Primer

“…Such indicators are overdue though, a global index, the net reclassification index (NRI; Pencina et al, ) is already available. Novel graphical techniques such as test validation plots (Neller & Frederick, ), predictiveness curves (Pepe et al, ), and predictive ROC curves (Shiu & Gatsonis, ) may also prove useful in portraying the calibration component of predictive validity across cut‐off thresholds.…”

Communicating the Risk of Violent and Offending Behavior: Review and Introduction to this Special Issue

“…Not surprisingly, poor numeracy or, more specifically, “risk illiteracy” (Garcia‐Retamero & Cokely, ) has emerged as an important individual difference factor that limits effectiveness of risk communication and decision‐making (e.g., Apter et al ., ; Fagerlin, Ubel, Smith, & Zikmund‐Fisher, ; Scurich, Monahan, & John, ; Waters et al ., ; Wegwarth & Gigerenzer, ). Communicating risk numerically can help to overcome problems of imprecision and poor reliability observed with non‐numeric communication (e.g., Neller & Frederick, ). Both laypeople and professionals, however, exhibit difficulty with numeric reasoning (e.g., Lipkus, Samsa, & Rimer, ), particularly with understanding relative risk (e.g., how much a treatment can reduce risk) and the number needed to treat in order to cure one person (e.g., Apter et al ., ; Bodemer et al ., ; Visschers, Meertens, Passchier, & de Vries, ).…”