Evidence for Hypodescent in Visual Semantic AI

Abstract

We examine the state-of-the-art multimodal "visual semantic" model CLIP ("Contrastive Language Image Pretraining") for the rule of hypodescent, or one-drop rule, whereby multiracial people are more likely to be assigned a racial or ethnic label corresponding to a minority or disadvantaged racial or ethnic group than to the equivalent majority or advantaged group. A face morphing experiment grounded in psychological research demonstrating hypodescent indicates that, at the midway point of 1, 000 series of morphed images, CLIP associates 69.7% of Black-White female images with a Black text label over a White text label, and similarly prefers Latina (75.8%) and Asian (89.1%) text labels at the midway point for Latina-White female and Asian-White female morphs, reflecting hypodescent. Additionally, assessment of the underlying cosine similarities in the model reveals that association with White is correlated with association with "person, " with Pearson's 𝜌 as high as 0.82, 𝑝 < 10 −90 over a 21, 000-image morph series, indicating that a White person corresponds to the default representation of a person in CLIP. Finally, we show that the stereotype-congruent pleasantness association of an image correlates with association with the Black text label in CLIP, with Pearson's 𝜌 = 0.48, 𝑝 < 10 −90 for 21, 000 Black-White multiracial male images, and 𝜌 = 0.41, 𝑝 < 10 −90 for Black-White multiracial female images. CLIP is trained on Englishlanguage text gathered using data collected from an American website (Wikipedia), and our findings demonstrate that CLIP embeds the values of American racial hierarchy, reflecting the implicit and explicit beliefs that are present in human minds. We contextualize these findings within the history of and psychology of hypodescent. Overall, the data suggests that AI supervised using natural language will, unless checked, learn biases that reflect racial hierarchies. CCS CONCEPTS• Computing methodologies → Artificial intelligence.

Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semantics

“…Charlesworth et al [15] showed that word embeddings concerning race/ethnic groups reveal stereotypes whose valence has remained stable for 200 years. Wolfe et al [52] demonstrated that racial bias reflecting the rule of hypodescent in multi-modal languagevision models is more prominent for women, assigning multiracial female images to the minority group's linguistic label. Wolfe and Caliskan [54] further showed that language-vision models mark the images of women in the language space due to deviation from the default representation of 'person. '…”