The Age of Secrecy and Unfairness in Recidivism Prediction

Abstract

and ultimately stagnating because there is no clear definition of fairness and competing definitions are largely incompatible. We argue that the focus on the question of fairness is misplaced, as these algorithms fail to meet a more important and yet readily obtainable goal: transparency. As a result, creators of secret algorithms can provide incomplete or misleading descriptions about how their models work, and various other kinds of errors can easily go unnoticed.By trying to partially reconstruct the COMPAS model -a recidivism-risk scoring model used throughout the criminal justice system -we show that it does not seem to depend linearly on the defendant's age, despite statements to the contrary by the model's creator. This observation has not been made before despite many recently published papers on this algorithm. Furthermore, by subtracting from COMPAS its (hypothesized) nonlinear age component, we show that COMPAS does not necessarily depend on race, contradicting ProPublica's analysis, which assumed linearity in age. In other words, faulty assumptions about a proprietary algorithm lead to faulty conclusions that go unchecked. Were the algorithm transparent in the first place, this likely would

Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead

“…But if the model is a black box, it is very difficult to manually calibrate how much this additional information should raise or lower the estimated risk. This issue arises constantly; for instance, the proprietary COMPAS model used in the U.S. Justice System for recidivism risk prediction does not depend on the seriousness of the current crime [27,29]. Instead, the judge is instructed to somehow manually combine current crime with COMPAS.…”

“…Typographical errors seem to be common in computing COMPAS, and these typographical errors sometimes determine bail decision outcomes [1,27]. This exemplifies an important drawback of using overly complicated black box models for recidivism prediction -they may be incorrectly calculated in practice.…”

“…COMPAS is not a machine learning model -it was not created by any standard machine learning algorithm. It was designed by experts based on carefully designed surveys and expertise, and it does not seem to depend heavily on past criminal history [27]. Interestingly, if the COMPAS model were not proprietary, its documentation [29] indicates that it would actually be an interpretable predictive model.…”

The Secrets of Machine Learning: Ten Things You Wish You Had Known Earlier to Be More Effective at Data Analysis

“…Since their model had a race coefficient that was statistically significantly different from zero according to standard regression analysis, they concluded that COMPAS depends on race, even after taking age and criminal history into account. However, their analysis had at least one serious flaw: COMPAS seems to depend non-linearly on age, and once that nonlinearity is considered and subtracted out, machine learning algorithms provide no evidence to back ProPublica's claim [47]. This shows one of the dangers of trying to explain black box models -the variables that are important for the explanation model (like race) are not always important to the black box.…”

Evaluating systematic transactional data enrichment and reuse

“…Prior studies in the ethics of machine learning have shown that without attention to algorithmic bias, that machine learning systems will contain bias inherent in the original training source [17,18]. Modern algorithms have also exhibited the problematic nature of reinforcing systematic bias intensifying many of the pressing social concerns of our era; such as poverty, racism, and the erosion of democracy [8,19,20]. While it is beyond the scope of this work to propose an alternative subject classification scheme for the universe of knowledge, this work attempts to undertake objective measures of the recommender systems by way of network science metrics; while at the same time underscoring here the inherent biases of the underlying subject data attached to books in the library.…”