The Emergence of Machine Learning Techniques in Criminology

Brennan, Tim; Oliver, W. L.

doi:10.1111/1745-9133.12055

Cited by 58 publications

(18 citation statements)

References 28 publications

(65 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As noted above, more advanced actuarial methods can be associated with loss of predictive validity when used beyond the initial development sample (Monahan et al, 2005). Although more recent studies of recidivism and health outcomes have achieved efficiency and avoided this “shrinkage” problem through bootstrapping (Duwe, 2012), applying stability analysis to classification trees (Berk & Bleich, 2014), using advanced modeling methods such as random forests (Barnes & Hyatt, 2012; Berk, Sherman, Barnes, Kurtz, & Ahlman, 2008), Bayesian networks (Constantinou, Freestone, Marsh, Fenton, & Coid, 2015), and lasso regression (Tse et al, 2015), or even developing predictors using multiple methods and then combining them through a superlearner process (Kreif, Grieve, Diaz, & Harrison, 2015), these techniques have not yet developed a wide following in the forensic disciplines (Brennan & Oliver, 2013). When (and if) they do, it would be important that their predictive validity can be compared rigorously with that of existing tools and newer versions of existing instruments such as HCR-20 Version 3 and Version 2 of OVP (Howard, 2015).…”

Section: Discussionmentioning

confidence: 99%

The Effect of Sample Heterogeneity and Risk Categorization on Area Under the Curve Predictive Validity Metrics

Howard

2016

Criminal Justice and Behavior

View full text Add to dashboard Cite

Forensic researchers often assume that the widely used area under the curve (AUC) predictive validity statistic can be readily compared across risk assessment instruments and between studies. From risk distributions for 224,771 convicted English and Welsh offenders, I quantify the extent to which the AUCs of two continuously scored actuarial instruments are dependent upon sample heterogeneity and risk categorization. Sample composition can cause predictive validity to vary by 10% to 20% over chance between subpopulations, with higher AUCs for female subgroups whose scores have high variability, and lower AUCs when sampling is restricted by index offense and/or sentence type. Risk categorization has a potentially large effect on AUCs, which fall when few categories are used, especially when those categories contain unequal numbers of offenders. An improved understanding of how these factors can affect AUCs will inform professionals using risk assessment instruments and researchers comparing the validity of multiple instruments.

show abstract

Section: Discussionmentioning

confidence: 99%

The Effect of Sample Heterogeneity and Risk Categorization on Area Under the Curve Predictive Validity Metrics

Howard

2016

Criminal Justice and Behavior

View full text Add to dashboard Cite

show abstract

“…However, there is a growing realization that, in addition to predictions, ML models are capable of producing knowledge about domain relationships contained in data, often referred to as interpretations. These interpretations have found uses in their own right, e.g., medicine (1), policymaking (2), and science (3,4), as well as in auditing the predictions themselves in response to issues such as regulatory pressure (5) and fairness (6). In these domains, interpretations have been shown to help with evaluating a learned model, providing information to repair a model (if needed), and building trust with domain experts (7).…”

mentioning

confidence: 99%

Definitions, methods, and applications in interpretable machine learning

Murdoch

Singh

Kumbier

et al. 2019

Proc. Natl. Acad. Sci. U.S.A.

1,271

723

View full text Add to dashboard Cite

Machine-learning models have demonstrated great success in learning complex patterns that enable them to make predictions about unobserved data. In addition to using models for prediction, the ability to interpret what a model has learned is receiving an increasing amount of attention. However, this increased focus has led to considerable confusion about the notion of interpretability. In particular, it is unclear how the wide array of proposed interpretation methods are related and what common concepts can be used to evaluate them. We aim to address these concerns by defining interpretability in the context of machine learning and introducing the predictive, descriptive, relevant (PDR) framework for discussing interpretations. The PDR framework provides 3 overarching desiderata for evaluation: predictive accuracy, descriptive accuracy, and relevancy, with relevancy judged relative to a human audience. Moreover, to help manage the deluge of interpretation methods, we introduce a categorization of existing techniques into model-based and post hoc categories, with subgroups including sparsity, modularity, and simulatability. To demonstrate how practitioners can use the PDR framework to evaluate and understand interpretations, we provide numerous real-world examples. These examples highlight the often underappreciated role played by human audiences in discussions of interpretability. Finally, based on our framework, we discuss limitations of existing methods and directions for future work. We hope that this work will provide a common vocabulary that will make it easier for both practitioners and researchers to discuss and choose from the full range of interpretation methods.interpretability | machine learning | explainability | relevancy

show abstract

“…The use of actuarial risk assessments in criminal justice settings has of late been subject to intense scrutiny. There have been ongoing discussions about how much better in practice risk assessments derived from machine learning perform compared to risk assessments derived from older, conventional methods (Berk 2012;Berk and Bleich 2013;Brennan and Oliver 2013;Liu et al 2011;Rhodes 2013;Ridgeway 2013aRidgeway , 2013b. We have learned that when relationships between predictors and the response are complex, machine learning approaches can perform far better.…”

mentioning

confidence: 99%

Fairness in Criminal Justice Risk Assessments: The State of the Art

Berk

Heidari

Jabbari

et al. 2018

Sociological Methods & Research

705

617

View full text Add to dashboard Cite

Objectives: Discussions of fairness in criminal justice risk assessments typically lack conceptual precision. Rhetoric too often substitutes for careful analysis. In this paper, we seek to clarify the tradeoffs between different kinds of fairness and between fairness and accuracy.Methods: We draw on the existing literatures in criminology, computer science and statistics to provide an integrated examination of fairness and accuracy in criminal justice risk assessments. We also provide an empirical illustration using data from arraignments.Results: We show that there are at least six kinds of fairness, some of which are incompatible with one another and with accuracy.Conclusions: Except in trivial cases, it is impossible to maximize accuracy and fairness at the same time, and impossible simultaneously to satisfy all kinds of fairness. In practice, a major complication is different base rates across different legally protected groups. There is a need to consider challenging tradeoffs.

show abstract

The Emergence of Machine Learning Techniques in Criminology

Cited by 58 publications

References 28 publications

The Effect of Sample Heterogeneity and Risk Categorization on Area Under the Curve Predictive Validity Metrics

The Effect of Sample Heterogeneity and Risk Categorization on Area Under the Curve Predictive Validity Metrics

Definitions, methods, and applications in interpretable machine learning

Fairness in Criminal Justice Risk Assessments: The State of the Art

Contact Info

Product

Resources

About