Evaluating Fairness of Algorithmic Risk Assessment Instruments: The Problem With Forcing Dichotomies

Zottola, Samantha A.; Desmarais, Sarah L.; Lowder, Evan M.; Clarke, Sarah Elizabeth Duhart

doi:10.1177/00938548211040544

Cited by 4 publications

(1 citation statement)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are numerous ways to determine a cutoff value (Kuhn & Johnson, 2013), but there is no agreed-upon approach to effectively determining which is the best method. Further, using a cutoff to enable the calculation of these fairness metrics is problematic, as different cutoffs will produce different fairness results (Zottola et al, 2022). Last, this study has treated predictive parity and error rate balance as equally important definitions of fairness to satisfy, which may not be reflective of forensic practice.…”

Section: Discussionmentioning

confidence: 99%

Statistical learning methods and cross-cultural fairness: Trade-offs and implications for risk assessment instruments.

Ashford¹,

Spivak²,

Ogloff³

et al. 2023

Psychological Assessment

View full text Add to dashboard Cite

The use of statistical learning methods has recently increased within the risk assessment literature. They have primarily been used to increase accuracy and the area under the curve (AUC, i.e., discrimination). Processing approaches applied to statistical learning methods have also emerged to increase cross-cultural fairness. However, these approaches are rarely trialed in the forensic psychology discipline nor have they been trialed as an approach to increase fairness in Australia. The study included 380 Aboriginal and Torres Strait Islander and non-Aboriginal and Torres Strait Islander males assessed with the Level of Service/Risk Needs Responsivity (LS/RNR). Discrimination was assessed through the AUC, and fairness was assessed through the cross area under the curve (xAUC), error rate balance, calibration, predictive parity, and statistical parity. Logistic regression, penalized logistic regression, random forest, stochastic gradient boosting, and support vector machine algorithms using the LS/RNR risk factors were used to compare performance against the LS/RNR total risk score. The algorithms were then subjected to pre- and postprocessing approaches to see if fairness could be improved. Statistical learning methods were found to produce comparable or marginally improved AUC values. Processing approaches increased several fairness definitions (namely xAUC, error rate balance, and statistical parity) between Aboriginal and Torres Strait Islanders and non-Aboriginal and Torres Strait Islanders. The findings demonstrate that statistical learning methods may be a useful approach to increasing the discrimination and cross-cultural fairness of risk assessment instruments. However, both fairness and the use of statistical learning methods encompass significant trade-offs that need to be considered.

show abstract

Section: Discussionmentioning

confidence: 99%