Missing the missing values: The ugly duckling of fairness in machine learning

Abstract

Nowadays, there is an increasing concern in machine learning about the causes underlying unfair decision making, that is, algorithmic decisions discriminating some groups over others, especially with groups that are defined over protected attributes, such as gender, race and nationality. Missing values are one frequent manifestation of all these latent causes: protected groups are more reluctant to give information that could be used against them, sensitive information for some groups can be erased by human operators, or data acquisition may simply be less complete and systematic for minority groups. However, most recent techniques, libraries and experimental results dealing with fairness in machine learning have simply ignored missing data. In this paper, we present the first comprehensive analysis of the relation between missing values and algorithmic fairness for machine learning: (1) we analyse the sources of missing data and bias, mapping the common causes, (2) we find that rows containing missing values are usually fairer than the rest, which should discourage the consideration of missing values as the uncomfortable ugly data that different techniques and libraries for handling algorithmic bias get rid of at the first occasion, (3) we studyThis is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

Fairness without Imputation: A Decision Tree Approach for Fair Prediction with Missing Values

“…In contrast, we focus on the case where the input features are missing and may thus impact performance in downstream prediction tasks. The works that are most relevant to our work are Fernando et al (2021); Wang and Singh (2021) as they examine the intersection of general data missingness and fairness. While Fernando et al (2021) presents a comprehensive investigation on the relationship between fairness and missing values, their analyses are limited to observational and empirical studies on how different ways of handling missing values can affect fairness.…”

“…The works that are most relevant to our work are Fernando et al (2021); Wang and Singh (2021) as they examine the intersection of general data missingness and fairness. While Fernando et al (2021) presents a comprehensive investigation on the relationship between fairness and missing values, their analyses are limited to observational and empirical studies on how different ways of handling missing values can affect fairness. Wang and Singh (2021) proposes reweighting scheme that assigns lower weight to data points with missing values by extending the preprocessing scheme given in Calmon et al (2017).…”

“…When there exist disparate missing patterns across different groups, how missing values are treated can affect the final trained prediction model, not just its accuracy, but also its fairness. Despite the ubiquity of the problem, fairness issues in missing values have gained only limited attention (Fernando et al 2021;Wang and Singh 2021). The burgeoning literature on fair ML algorithms either overlooks the existence of missing data or assumes that the training data are always complete (see e.g., Calmon et al 2017;Hardt, Price, and Srebro 2016;Zemel et al 2013;Zafar et al 2019;Feldman et al 2015;Menon and Williamson 2018).…”

Interpretable Data-Based Explanations for Fairness Debugging

“…This means that if the ML algorithms had been trained on the modified training data, it would not have exhibited the unexpected or undesirable behavior or would have exhibited this behavior to a lesser degree. Explanations generated by our framework, which complement existing approaches in XAI, are crucial for helping system developers and ML practitioners to debug ML algorithms for data errors and bias in training data, such as measurement errors and misclassifications [35,42,94], data imbalance [27], missing data and selection bias [29,62,63], covariate shift [74,82], technical biases introduced during data preparation [85], and poisonous data points injected through adversarial attacks [36,43,65,83]. It is known in the algorithmic fairness literature that information about the source of bias is critically needed to build fair ML algorithms because no current bias mitigation solution fits all situations [27,31,36,82,94].…”

“…More compact and coherent descriptions are needed. Furthermore, sources of bias and discrimination in training data are typically not randomly distributed across different sub-populations; rather they manifest systematic errors in data collection, selection, feature engineering, and curation [29,35,42,62,63,70,94]. That is, more often than not, certain cohesive subsets of training data are responsible for bias.…”