A Step Toward More Inclusive People Annotations for Fairness

Abstract

The Open Images Dataset [16] contains approximately 9 million images and is a widely accepted dataset for computer vision research. As is common practice for large datasets, the annotations are not exhaustive, with bounding boxes and attribute labels for only a subset of the classes in each image. In this paper, we present a new set of annotations on a subset of the Open Images dataset called the MIAP (More Inclusive Annotations for People) subset, containing bounding boxes and attributes for all of the people visible in those images. The attributes and labeling methodology for the MIAP subset were designed to enable research into model fairness.In addition, we analyze the original annotation methodology for the person class and its subclasses, discussing the resulting patterns in order to inform future annotation efforts. By considering both the original and exhaustive annotation sets, researchers can also now study how systematic patterns in training annotations affect modeling. CCS CONCEPTS• Computing methodologies → Computer vision; Machine learning; • Social and professional topics → User characteristics.

Fairness Indicators for Systematic Assessments of Visual Feature Extractors

“…Several fairness evaluation datasets have been proposed to facilitate fairness assessment by enabling testing of classification performance on images from diverse geographic locations [70] or correlation between detection performance and an income variable of the object [11]. Recent work emphasized the importance of how people images are classified or otherwise analyzed by computer vision systems from early datasets of faces with geographically diverse collection [44,53] or Buolamwini and Gebru [5]'s intersectional benchmark to the recent datasets FairFace [39], Casual Conversations [30] and More Inclusive Images for People (MIAP) [68]. These works offer curated datasets with labels obtained through clear annotation rules and with specific efforts deployed for checking annotation bias.…”

“…We describe the datasets we use in Table 1: Casual Conversations [30], OpenImages MIAP [68], and UTK Faces [79] contain images of people and are used in the indicators of harmful label association and/or the same-group similarity search. DollarStreet [11,23], is used in the geographical fairness indicator.…”

“…first, training attribute specific classifiers (such as gender, age, skintone etc) is increasingly contrary to intended uses of datasets collected for fairness, [30,68],…”

“…OpenImages MIAP [68] is a dataset designed to enable ML Fairness and constructed by providing additional annotations for a subset of OpenImages v6 [49] dataset. These annotations include bounding boxes of people and attribute labels for fairness such as perceived gender (predominantly masculine, predominantly feminine, unknown) presentation and perceived age range (young, middle, older, unknown).…”

“…To this end, we propose three fairness indicators, which aim at quantifying specific harms and biases for certain image based computer vision feature extractors. Our proposed fairness indicators use publicly available datasets previously collected to measure fairness in computer vision [23,30,68,79], and focus on systematically evaluating three main sources of harms that have been identified in the literature:…”