Leave-one-out Unfairness

Abstract

We introduce leave-one-out unfairness, which characterizes how likely a model's prediction for an individual will change due to the inclusion or removal of a single other person in the model's training data. Leave-one-out unfairness appeals to the idea that fair decisions are not arbitrary: they should not be based on the chance event of any one person's inclusion in the training data. Leave-one-out unfairness is closely related to algorithmic stability, but it focuses on the consistency of an individual point's prediction outcome over unit changes to the training data, rather than the error of the model in aggregate. Beyond formalizing leave-one-out unfairness, we characterize the extent to which deep models behave leave-one-out unfairly on real data, including in cases where the generalization error is small. Further, we demonstrate that adversarial training and randomized smoothing techniques have opposite effects on leave-one-out fairness, which sheds light on the relationships between robustness, memorization, individual fairness, and leave-one-out fairness in deep models. Finally, we discuss salient practical applications that may be negatively affected by leave-one-out unfairness.

Consistent Counterfactuals for Deep Models

“…While in this work, we address the problem of multiplicitious deep models producing varying outputs on counterfactual examples, recent work has shown that there are large differences in model prediction behavior on any input across small changes to the model (Black & Fredrikson, 2021;Marx et al, 2019;D'Amour et al, 2020). Instability has also been shown to be a problem for gradient-based explanations, although this is largely studied in an adversarial context (Dombrowski et al, 2019;Ghorbani et al, 2019;Heo et al, 2019).…”

Selective Ensembles for Consistent Predictions

“…However, preventing inconsistent predictions and abstaining from uncertain predictions are different goals: in our setting, the aim is to predict the mode across models drawn from a certain distribution, whereas calibration is measured against predicting the true label. Moreover, prior work has shown that confidence scores may not be correlated with prediction consistency across models with different random initializations (Black and Fredrikson, 2021). Finally, while abstaining on points with low confidence scores may lead to greater consistency, it may not yield a guarantee, which this work provides.…”

“…As the initial parameters of the model tend to be determined by a random seed, we will interchangeably refer to this as the selection of random seed. More generally, both of these types of choices instantiate a broader class of choices that could be considered arbitrary, despite the fact that they may impact the predictions (Black and Fredrikson, 2021;Marx et al, 2019;Mehrer et al, 2020) (Section 5.1) and explanations (Section 5.2) of the resulting model.…”

“…Prior work has shown that deep models are inconsistent in their predictions across arbitrary random changes in their training pipeline, such as initialization parameters and makeup of the training set (Black and Fredrikson, 2021;Mehrer et al, 2020;D'Amour et al, 2020;Kolen and Pollack, 1991;Feldman, 2019). The problem of model sensitivity, particularly to variability in the training set, can lead to an increase generalization error (Elisseeff et al, 2003) as well as to leaking training set information (Dwork, 2006;Yeom et al, 2018).…”

“…Recent work has drawn attention to the fact that models that appear similar from aggregate quality measures, such as accuracy, often have markedly different behavior at the level of individual predictions (Black and Fredrikson, 2021;Marx et al, 2019). Further, in deep models, this inconsistency can arise even between closely-related models, such as those arising from different initializations, or from leave-one-out differences in the training data (Black and Fredrikson, 2021;D'Amour et al, 2020;Mehrer et al, 2020). This behavior is undesirable in many high-stakes contexts, such as medical applications and credit-approving scenarios, as it may cast doubt on the justifiability of the model's outcome and pose difficulties for reproducibility and comparison.…”