Diversity and Inclusion Metrics in Subset Selection

Abstract

The ethical concept of fairness has recently been applied in machine learning (ML) settings to describe a wide range of constraints and objectives. When considering the relevance of ethical concepts to subset selection problems, the concepts of diversity and inclusion are additionally applicable in order to create outputs that account for social power and access differentials. We introduce metrics based on these concepts, which can be applied together, separately, and in tandem with additional fairness constraints. Results from human subject experiments lend support to the proposed criteria. Social choice methods can additionally be leveraged to aggregate and choose preferable sets, and we detail how these may be applied. CCS CONCEPTS• Information systems → Information retrieval diversity; Evaluation of retrieval results. KEYWORDS machine learning fairness, subset selection, diversity and inclusion ACM Reference Format:

Gender Bias in Machine Translation

“…This is a value-laden option(Birhane et al, 2020), and not the only possible one(Mitchell et al, 2020). For a broader discussion on measurement and bias we refer the reader also to Jacobs (2021);Jacobs et al (2020).…”

Diversity by Design in Music Recommender Systems

“…The conceptualisation and measurement of diversity have been the object of study of a broad range of disciplines (Stirling, 2007;Steel et al, 2018;Mitchell et al, 2020), and to identify a global framework for measuring music recommendation diversity is out of the scope of this overview. Several diversity indexes have been formulated to describe different kinds of populations, among which most of them fall within the category of the so-called dual-concept diversity (McDonald and Dimmick, 2003).…”

‘The names have changed, but the game’s the same’: artificial intelligence and racial policy in the USA

“…Dressel and Farid [5], Mitchell et al [16], and many others view algorithmic instability, uncertainty, and imprecision as defects to be corrected by the exercise of externallyimposed control. Mitchell et al put it thus:…”