Color-Theoretic Experiments to Understand Unequal Gender Classification Accuracy From Face Images

Abstract

Recent work shows unequal performance of commercial face classification services in the gender classification task across intersectional groups defined by skin type and gender. Accuracy on dark-skinned females is significantly worse than on any other group. In this paper, we conduct several analyses to try to uncover the reason for this gap. The main finding, perhaps surprisingly, is that skin type is not the driver. This conclusion is reached via stability experiments that vary an image's skin type via colortheoretic methods, namely luminance mode-shift and optimal transport. A second suspect, hair length, is also shown not to be the driver via experiments on face images cropped to exclude the hair. Finally, using contrastive post-hoc explanation techniques for neural networks, we bring forth evidence suggesting that differences in lip, eye and cheek structure across ethnicity lead to the differences. Further, lip and eye makeup are seen as strong predictors for a female face, which is a troubling propagation of a gender stereotype.

Towards Causal Benchmarking of Bias in Face Analysis Algorithms

“…This is not easy in the case of image data, leading us to ask the question: Can one systematically measure bias in computer vision algorithms using the experimental method? While this is not immediately intuitive [56], we find that the answer is yes, and offer a practical way forward.…”

“…However, the evaluations are based on observational rather than interventional techniques -and therefore any conclusions from these studies should be treated with caution. A notable exception is a recent study [56] using the experimental method to investigate the effect of skin color in gender classification. In that study, skin color is modified artificially in photographs of real faces to measure the effects of differences in skin color, all else being equal.…”

“…For example, the appearance of the parliamentarians in the PPB dataset [12] tends to be gender-stereotypical, e.g., very few males have long hair and almost no light-skinned females have short hair (Fig. 12, and [56]). The fact that hair length (a variable that may affect gender classification accuracy) is correlated in PPB with skin color (a variable of interest) complicates the analysis.…”

Face recognition vendor test part 3:

“…Note that manifestly poor (and unrecognizable) images can be collected from mis-configured cameras, without any algorithmic or AI culpability. Indeed, after publication of the MIT studies [5,36] on bias in gender-estimation algorithms, suspicion fell upon the presence of poor [30,31] -in short, "skin type by itself has a minimal effect on the classification decision".…”

AI and recruiting software: Ethical and legal implications

“…Whereas for other facial analysis, the categories are known beforehand: one-to-one matching in face detection, two genders and a closed list of six universal emotion types [8]. The lack of accuracy in a subgroup of dark-skinned women stands out in the study by Muthukumar et al [20]. Other authors [21,22] have noted the difficulties in face detection and race in unconstrained captures, especially the blurred ones.…”