Pretrial risk assessment validation research: Range restriction and attenuation of predictive validity estimates.

Abstract

Objectives: A growing number of studies have examined the predictive validity of pretrial risk assessments. Overwhelmingly, these validation studies are conducted in the context of routine practice, where not all individuals who are assessed receive pretrial release. Despite evidence of range restriction in pretrial validation research, no study to date has corrected for range restriction in predictive validity estimates. To address this limitation, we examined the effects of range restriction on predictive validity estimates under varying conditions. Method: We ran simulations based on data from a local validation of 1,030 pretrial defendants to demonstrate the effects of range restriction on predictive validity estimates under different degrees of range restriction, various population correlations, and with and without the influence of a third variable, z, representing discretionary decision making. We examined the effect of range restriction on correlation coefficient (r) and nonparametric Area Under the Curve (AUC) statistics. Results: Under a realistic population correlation (q = .40), attenuation of r ranged from 1-29% for total scores and 8-39% for risk levels across conditions. Under similar conditions, attenuation for AUC estimates ranged from 1-13% for total scores and 1-20% for risk levels. Attenuation was greater with the influence of a secondary selection variable modeling discretionary decision making above and beyond the risk assessment tool. Conclusion: Range restriction may meaningfully reduce predictive validity estimates when greater than 40% of moderate risk and 60% of high risk defendants are detained. Public Significance StatementIncreasingly, jurisdictions are looking to validate pretrial risk assessments used in practice. Risk assessments may be less accurate in predicting outcomes when a substantial proportion of moderateand high-risk defendants are detained pending trial. Future studies should report on the proportion of assessed defendants detained pretrial and correct findings for potential loss of observations.

Pretrial risk assessment instruments in practice: The role of judicial discretion in pretrial reform

“…However, others have indicated that AUCs less than 0.7 provide poor discrimination (Mandrekar, 2010), suggesting that the threshold used in the field of criminal justice may be somewhat biased in favor of risk assessment tools. 14 Recent research suggests that range restriction, via the detention of a high-risk individuals, may reduce predictive validity estimates (Lowder & Wilson, 2021). Although the share of high-risk defendants detained in the current sample (49%) fell below the threshold of concern identified in this prior work, we nevertheless recognize the potential influence of range restriction on our predictive validity estimates.…”

“…1414Recent research suggests that range restriction, via the detention of a high‐risk individuals, may reduce predictive validity estimates (Lowder & Wilson, 2021). Although the share of high‐risk defendants detained in the current sample (49%) fell below the threshold of concern identified in this prior work, we nevertheless recognize the potential influence of range restriction on our predictive validity estimates.…”

Comparing the relationships between money bail, pretrial risk scores, and pretrial outcomes.

“…There have been concerns that this practice results in a restriction of the range of risk scores and therefore affects the predictive validity of risk assessment instruments. However, recent work suggests that more than 40% of people with moderate scores and 60% of people with high-risk scores in a sample must be detained before range restriction meaningfully impacts prediction (Lowder & Wilson, 2021). We are well under this cutoff, given that fewer than 5% of the people in our total sample were detained for the entire pretrial period.…”