Valuing Structured Professional Judgment: Predictive Validity, Decision‐making, and the Clinical‐Actuarial Conflict

Abstract

Structured professional judgment (SPJ) has received considerable attention as an alternative to unstructured clinical judgment and actuarial assessment, and as a means of resolving their ongoing conflict. However, predictive validity studies have typically relied on receiver operating characteristic (ROC) analysis, the same technique commonly used to validate actuarial assessment tools. This paper presents SPJ as distinct from both unstructured clinical judgment and actuarial assessment. A key distinguishing feature of SPJ is the contribution of modifiable factors, either dynamic or protective, to summary risk ratings. With modifiable factors, the summary rating scheme serves as a prognostic model rather than a classification procedure. However, prognostic models require more extensive and thorough predictive validity testing than can be provided by ROC analysis. It is proposed that validation should include calibration and reclassification techniques, as well as additional measures of discrimination. Several techniques and measures are described and illustrated. The paper concludes by tracing the limitations of ROC analysis to its philosophical foundation and its origin as a statistical theory of decision-making. This foundation inhibits the performance of crucial tasks, such as determining the sufficiency of a risk assessment and examining the evidentiary value of statistical findings. The paper closes by noting a current effort to establish a viable and complementary relationship between SPJ and decision-making theory.

The Predictive Validity of the Structured Assessment of Violence Risk in Youth for Young Spanish Offenders

“…The ROC analysis is especially indicated in cases where a decision must be made, where it is essential to know in detail how accurate the different diagnostic tests are, and whether they correctly classify patients in categories or conditions related to a certain criterion (Fazel, 2013; Mossman, 2013). The total and partial scores of the instrument present statistically significant correlations different from zero and greater than 0.34 between the scores and factors of the SAVRY and recurrence of the S-ASB.…”

Risk Assessment and Manage- ment of Violence in Patients with Mental Disorders: A Re- view

“…In addition, SPJ allows evaluators to consider case specific factors, and to modify the overall level of risk. This format allows evaluators to recognize how the assessment informs an intervention to reduce the risk of violence [34]. SPJ may therefore be superior to actuarial risk assessment in terms of clinical feasibility.…”

Needle in a Haystack

“…Ample research has established the superiority of actuarial prediction over clinical discretion, but equally rigorous research has shown how actuarial instruments “black box” subjective and discretionary decisions (Falzer, ; Hannah‐Moffat, Maurutto, and Turnbull, ), which means that the information and process used to score the criteria itemized in each risk or need category is not consistent or transparent to anyone but the assessor. The only thing provided is the score or priorities produced by the assessment tool.…”

“…Although the science behind them is known to be imprecise, actuarial assessments are routinely adopted and rigorously defended. A polarized debate has emerged between those who promote actuarial assessments and their continued refinement and those who question the epistemological basis of this practice and/or advocate for blended models that integrate empirically structured assessment and discretion (Falzer, ; Hart and Cooke, ; Neller and Frederick, ; Rossegger et al., ) . Within this context, researchers and practitioners recognize the complexity involved in issues such as treatment, desistance, and recidivism.…”