Effects of Automating Recidivism Risk Assessment on Reliability, Predictive Validity, and Return on Investment (ROI)

Abstract

Research Summary The relationship between reliability and validity is an important but often overlooked topic of research on risk assessment tools in the criminal justice system. By using data from the Minnesota Screening Tool Assessing Recidivism Risk (MnSTARR), a risk assessment instrument the Minnesota Department of Corrections (MnDOC) developed and began using in 2013, we evaluated the impact of inter‐rater reliability (IRR) on predictive performance (validity) among offenders released in 2014. After comparing the reliability of a manual scoring process with an automated one, we found the MnSTARR was scored with a high degree of consistency by MnDOC staff as intraclass correlation (ICC) values ranged from 0.81 to 0.94. But despite this level of IRR, we still observed a degradation in predictive validity given that automated assessments significantly outperformed those that had been scored manually. Additional analyses revealed that the more inter‐rater disagreement increased, the more predictive performance decreased. The results from our cost–benefit analyses, which examined the anticipated impact of the MnDOC's efforts to automate the MnSTARR, showed that for every dollar to be spent on automation, the estimated return will be at least $4.35 within the first year and as much as $21.74 after the fifth year. Policy Implications Although it is unclear the degree to which our findings, which are somewhat preliminary, are generalizable to other offender populations and correctional systems, we believe the results are sufficiently promising to warrant greater interest in automating the assessment of risk and need. We anticipate many, if not most, correctional systems may need to invest in upgrading their IT infrastructure to support the use of automated instruments. But we also anticipate this investment would deliver a favorable return for our results suggest that automation reduces inter‐rater disagreement, which in turn improves predictive performance. Even if automation did not improve performance, the increased efficiency it produces would create reinvestment opportunities within correctional systems.

Automating Risk Assessment Instruments and Reliability

“…The advent of actuarial risk assessment instruments has given rise to a rich literature demonstrating the superiority of these tools at predicting recidivism over traditional approaches involving professional or clinical judgment (Andrews and Bonta, ; Andrews, Bonta, and Wormith, ; Bonta, Law, and Hanson, ). As discussed by Grant Duwe and Michael Rocque (, this issue), however, the aim of much of this literature has been focused on issues of validity or on the predictive power of these instruments (Gendreau, Goggin, and Smith, ; Gendreau, Little, and Goggin, ; Smith, Cullen, and Latessa, ). There have been fewer efforts to examine inter‐rater reliability or the extent to which raters generate consistent scores across assessments (Desmarais and Singh, ; Rocque and Plummer‐Beale, ) .…”

“…There have been fewer efforts to examine inter‐rater reliability or the extent to which raters generate consistent scores across assessments (Desmarais and Singh, ; Rocque and Plummer‐Beale, ) . Moreover, no studies to date have been conducted to examine the relationship between reliability and validity and the capacity of automated machine scoring, where risk instruments are populated through electronic data extraction methods, to minimize reliability problems and enhance prediction (Duwe and Rocque, ). This policy essay will discuss the importance of the study by Duwe and Rocque to the literature on reliability and automated scoring, identify potential areas of future research, and highlight the policy implications inherent in assessing reliability and using automated scoring techniques.…”

Automating Recidivism Risk Assessment

“…Because the methods of scoring and the instruments themselves vary widely, research is needed on the approaches that lead to the most reliable and valid outcomes. As a response to this need, Grant Duwe and Michael Rocque (, this issue) study the relationship between reliability and validity with data from the Minnesota Screening Tool Assessing Recidivism Risk (MnSTARR), a risk assessment instrument the Minnesota Department of Corrections (MnDOC) developed and began using in 2013. By using follow‐up data on offenders released in 2014 and manual MnSTARR assessments scored by MnDOC staff, Duwe and Rocque assess the impact of inter‐rater reliability (IRR) on predictive performance (validity).…”

“…Duwe and Rocque () find the MnSTARR was scored with a high degree of consistency by MnDOC staff and that intraclass correlation (ICC) values were at high levels, which they attribute to the instrument comprising mostly objective rather than subjective risk factors. Even with high IRR values on the manually scored instruments, they also report that (a) the automated assessments significantly outperformed those that had been scored manually and, as might be expected, (b) that the more inter‐rater disagreement increased, the more predictive performance decreased.…”

“…The two scholars commenting on the Duwe and Rocque () article are accomplished researchers in recidivism risk assessment studies. Thomas Cohen (, this issue) states that the predictive efficiency of recidivism risk assessment instruments “is of paramount importance because of the sheer numbers of persons currently placed under some form of correctional supervision and their proclivity to reoffend.” Stephen Wormith (, this issue) observes that the “research and development of the MnSTARR 2.0 by Duwe and Rocque () represents the confluence of many different kinds of efforts that have accumulated over the last 40 years to improve the science and practice of offender risk assessment.” These authors also describe various important aspects of this line of research and its contributions to corrections and recidivism research and practice.…”