To Predict and Serve?

Lum, Kristian; Isaac, William

doi:10.1111/j.1740-9713.2016.00960.x

Cited by 400 publications

(323 citation statements)

References 13 publications

Supporting

Mentioning

303

Contrasting

Order By: Relevance

“…Second, the chosen target variable might be measured less accurately for certain groups. For example, arrests are often used as a proxy for crime in applications of machine learning to policing and criminal justice, even though arrests are a racially biased representation of the true incidence of crime [28]. In treating arrests as a reliable proxy for crime, the model learns to replicate the biased labels in its predictions.…”

Section: Introductionmentioning

confidence: 99%

Problem Formulation and Fairness

Passi

Barocas

2019

Proceedings of the Conference on Fairness, Accountability, and Transparency

196

118

View full text Add to dashboard Cite

Formulating data science problems is an uncertain and difficult process. It requires various forms of discretionary work to translate high-level objectives or strategic goals into tractable problems, necessitating, among other things, the identification of appropriate target variables and proxies. While these choices are rarely self-evident, normative assessments of data science projects often take them for granted, even though different translations can raise profoundly different ethical concerns. Whether we consider a data science project fair often has as much to do with the formulation of the problem as any property of the resulting model. Building on six months of ethnographic fieldwork with a corporate data science team-and channeling ideas from sociology and history of science, critical data studies, and early writing on knowledge discovery in databases-we describe the complex set of actors and activities involved in problem formulation. Our research demonstrates that the specification and operationalization of the problem are always negotiated and elastic, and rarely worked out with explicit normative considerations in mind. In so doing, we show that careful accounts of everyday data science work can help us better understand how and why data science problems are posed in certain ways-and why specific formulations prevail in practice, even in the face of what might seem like normatively preferable alternatives. We conclude by discussing the implications of our findings, arguing that effective normative interventions will require attending to the practical work of problem formulation.

show abstract

Section: Introductionmentioning

confidence: 99%

Problem Formulation and Fairness

Passi

Barocas

2019

Proceedings of the Conference on Fairness, Accountability, and Transparency

196

118

View full text Add to dashboard Cite

show abstract

“…Training data often reflect societal biases and are not representative of the population (sample bias). Moreover, system bias might lead into generation of biased data which result into biased models that further reinforce such discriminatory policies, like in predictive policing [4].…”

Section: Introductionmentioning

confidence: 99%

FAE: A Fairness-Aware Ensemble Framework

Iosifidis

Fetahu

Ntoutsi

2019

2019 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Automated decision making based on big data and machine learning (ML) algorithms can result in discriminatory decisions against certain protected groups defined upon personal data like gender, race, sexual orientation etc. Such algorithms designed to discover patterns in big data might not only pick up any encoded societal biases in the training data, but even worse, they might reinforce such biases resulting in more severe discrimination. The majority of thus far proposed fairness-aware machine learning approaches focus solely on the pre-, in-or post-processing steps of the machine learning process, that is, input data, learning algorithms or derived models, respectively. However, the fairness problem cannot be isolated to a single step of the ML process. Rather, discrimination is often a result of complex interactions between big data and algorithms, and therefore, a more holistic approach is required.The proposed FAE (Fairness-Aware Ensemble) framework combines fairness-related interventions at both pre-and postprocessing steps of the data analysis process. In the preprocessing step, we tackle the problems of under-representation of the protected group (group imbalance) and of class-imbalance by generating balanced training samples. In the post-processing step, we tackle the problem of class overlapping by shifting the decision boundary in the direction of fairness.

show abstract

“…An experiment conducted by the Human Rights Data Analysis Group shows what this might look like. Applying a simulation of PredPol's algorithm to drug offences in Oakland, California, the study demonstrated that the algorithm would send officers mostly to African American and Latino neighbourhoods, despite evidence from the US 2011 National Survey on Drug Use and Health that drug use is dispersed evenly across all of Oakland's neighbourhoods 9 .…”

Section: The Future Of Predictionmentioning

confidence: 98%