A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle

Abstract

As machine learning (ML) increasingly affects people and society, awareness of its potential unwanted consequences has also grown. To anticipate, prevent, and mitigate undesirable downstream consequences, it is critical that we understand when and how harm might be introduced throughout the ML life cycle. In this paper, we provide a framework that identifies seven distinct potential sources of downstream harm in machine learning, spanning data collection, development, and deployment. In doing so, we aim to facilitate more productive and precise communication around these issues, as well as more direct, application-grounded ways to mitigate them.

Bias in Automated Speaker Recognition

“…We draw on Suresh and Guttag's [49] framework for understanding sources of harm through the machine learning life cycle to ground our investigation into bias in automated speaker recognition. Suresh and Guttag divide the machine learning life cycle into two streams and identify seven sources of bias related harms across the two streams: 1) the data generation stream can contain historical, representational and measurement bias; and 2) the model building and implementation stream can contain learning, aggregation, evaluation and deployment bias.…”

“…Representative benchmark datasets are particularly important during machine learning development, as benchmarks have disproportionate power to scale bias across applications if models overfit to the data in the benchmark [49]. Three evaluation sets can be constructed from the VoxCeleb 1 dataset to benchmark speaker verification models.…”

“…Drawing on Suresh and Guttag's Framework for Understanding Sources of Harm [49], we present the first detailed study on bias in the speaker recognition domain. We approach this work as a combination of an analytical and empirical evaluation focused on the VoxCeleb Speaker Recognition Challenge [31], one of the most popular benchmarks in the domain with widely used datasets.…”

SoK: Machine Learning Governance

“…Unmitigated, these biases transfer into algorithmic decisions that are fed back into the decision making process, thus aggravating the situation. Furthermore, algorithmic decisions themselves can also encode new biases that were not initially present in historical data [72]; this may be due to poor quality [73], and even the careless use of fairness constraints. [74] Like integrity and privacy, the challenges of algorithmic fairness can be considered through the lens of attacks and defenses.…”

“…For example, in pre-trials where a judge has to assess the risk of accepting or rejecting bail, the defendants are only being subject to algorithmic predictions because they have been arrested in the first place. Mitchell et al [75] provide a broad overview on various fairness risks associated with three broad categories: (i) data bias: both statistical-which evolves from lack of representation (i.e., selective labels [76]), and societal-which indicates a normative mismatch [73]; (ii) model bias: influence of choice of hypothesis [77] and even interpretability, and (iii) evaluation: how today's predictions can alter decision subjects' behaviors, thus requiring new predictions in the future. We will discuss such performative qualities of models in § III-D1.…”