Making fair ML software using trustworthy explanation

Chakraborty, Joymallya; Peng, Kewen

doi:10.1145/3324884.3418932

Cited by 26 publications

(12 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Baseline: We used a logistic regression model for creating baseline results. Logistic regression is widely used in the fairness domain as baseline model [36,[42][43][44]68]. We used scikit-learn implementation with 'l2' regularization, 'lbfgs' solver, and maximum iteration of 1000.…”

Section: Methodology 31 Modelsmentioning

confidence: 99%

Fair Enough: Searching for Sufficient Measures of Fairness

Majumder¹,

Chakraborty²,

Bai³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Testing machine learning software for ethical bias has become a pressing current concern. In response, recent research has proposed a plethora of new fairness metrics, for example, the dozens of fairness metrics in the IBM AIF360 toolkit. This raises the question: How can any fairness tool satisfy such a diverse range of goals?While we cannot completely simplify the task of fairness testing, we can certainly reduce the problem. This paper shows that many of those fairness metrics effectively measure the same thing. Based on experiments using seven real-world datasets, we find that (a) 26 classification metrics can be clustered into seven groups, and (b) four dataset metrics can be clustered into three groups. Further, each reduced set may actually predict different things. Hence, it is no longer necessary (or even possible) to satisfy all fairness metrics.In summary, to simplify the fairness testing problem, we recommend the following steps: (1) determine what type of fairness is desirable (and we offer a handful of such types); then (2) lookup those types in our clusters; then (3) just test for one item per cluster. To support that processing, all our scripts (and example datasets) are available at https://github.com/Repoanonymous/ Fairness_Metrics. KEYWORDSEthics in Software Engineering, Machine Learning with and for SE 1 Note: Here, by "insensitive" clusters, we mean those where changes to the data do not change the fairness scores.

show abstract

Section: Methodology 31 Modelsmentioning

confidence: 99%

Fair Enough: Searching for Sufficient Measures of Fairness

Majumder¹,

Chakraborty²,

Bai³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Previous studies showed that certain features contribute more to the predictive quality of the model [28,56]. Feature importance in prediction and corelation of features with the sensitive attribute also led to bias detection [16,31] in ML models. However, does creating new features (by removing certain semantics) from a potentially biased feature increase the fairness, is an open question.…”

Section: Fairness Analysis Of Stagesmentioning

confidence: 99%

Fair preprocessing: towards understanding compositional fairness of data transformers in machine learning pipeline

Biswas

Rajan

2021

Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

In recent years, many incidents have been reported where machine learning models exhibited discrimination among people based on race, sex, age, etc. Research has been conducted to measure and mitigate unfairness in machine learning models. For a machine learning task, it is a common practice to build a pipeline that includes an ordered set of data preprocessing stages followed by a classifier. However, most of the research on fairness has considered a single classifier based prediction task. What are the fairness impacts of the preprocessing stages in machine learning pipeline? Furthermore, studies showed that often the root cause of unfairness is ingrained in the data itself, rather than the model. But no research has been conducted to measure the unfairness caused by a specific transformation made in the data preprocessing stage. In this paper, we introduced the causal method of fairness to reason about the fairness impact of data preprocessing stages in ML pipeline. We leveraged existing metrics to define the fairness measures of the stages. Then we conducted a detailed fairness evaluation of the preprocessing stages in 37 pipelines collected from three different sources. Our results show that certain data transformers are causing the model to exhibit unfairness. We identified a number of fairness patterns in several categories of data transformers. Finally, we showed how the local fairness of a preprocessing stage composes in the global fairness of the pipeline. We used the fairness composition to choose appropriate downstream transformer that mitigates unfairness in the machine learning pipeline. CCS CONCEPTS• Software and its engineering → Software creation and management; • Computing methodologies → Machine learning.

show abstract

“…Table 1 shows seven fairness datasets used in this work. These datasets are very popular in the fairness domain and have been used by many prior researchers [9,[11][12][13][14]. All of these datasets contain at least one protected attribute.…”

Section: Fairness In ML Softwarementioning

confidence: 99%

xFAIR: Better Fairness via Model-based Rebalancing of Protected Attributes

Peng

Chakraborty

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning software can generate models that inappropriately discriminate against specific protected social groups (e.g., groups based on gender, ethnicity, etc). Motivated by those results, software engineering researchers have proposed many methods for mitigating those discriminatory effects.While those methods are effective in mitigating bias, few of them can provide explanations on what is the cause of bias. Here we propose xFAIR, a model-based extrapolation method, that is capable of both mitigating bias and explaining the cause. In our xFAIR approach, protected attributes are represented by models learned from the other independent variables (and these models offer extrapolations over the space between existing examples). We then use the extrapolation models to relabel protected attributes, which aims to offset the biased predictions of the classification model via rebalancing the distribution of protected attributes.The experiments of this paper show that, without compromising (original) model performance, xFAIR can achieve significantly better group and individual fairness (as measured in different metrics) than benchmark methods. Moreover, when compared to another instance-based rebalancing method, our model-based approach shows faster runtime and thus better scalability.For replication purposes, we make our algorithm available at https://github.com/anonymous12138/biasmitigation.

show abstract

Making fair ML software using trustworthy explanation

Cited by 26 publications

References 31 publications

Fair Enough: Searching for Sufficient Measures of Fairness

Fair Enough: Searching for Sufficient Measures of Fairness

Fair preprocessing: towards understanding compositional fairness of data transformers in machine learning pipeline

xFAIR: Better Fairness via Model-based Rebalancing of Protected Attributes

Contact Info

Product

Resources

About