Definition drives design: Disability models and mechanisms of bias in AI technologies

Abstract

The increasing deployment of artificial intelligence (AI) tools to inform decision-making across diverse areas including healthcare, employment, social benefits, and government policy, presents a serious risk for disabled people, who have been shown to face bias in AI implementations. While there has been significant work on analysing and mitigating algorithmic bias, the broader mechanisms of how bias emerges in AI applications are not well understood, hampering efforts to address bias where it begins. In this article, we illustrate how bias in AI-assisted decision-making can arise from a range of specific design decisions, each of which may seem self-contained and non-biasing when considered separately. These design decisions include basic problem formulation, the data chosen for analysis, the use the AI technology is put to, and operational design elements in addition to the core algorithmic design. We draw on three historical models of disability common to different decision-making settings to demonstrate how differences in the definition of disability can lead to highly distinct decisions on each of these aspects of design, leading in turn to AI technologies with a variety of biases and downstream effects. We further show that the potential harms arising from inappropriate definitions of disability in fundamental design stages are further amplified by a lack of transparency and disabled participation throughout the AI design process. Our analysis provides a framework for critically examining AI technologies in decision-making contexts and guiding the development of a design praxis for disability-related AI analytics. We put forth this article to provide key questions to facilitate disability-led design and participatory development to produce more fair and equitable AI technologies in disability-related contexts.

Augmented Cognition: Life as we don't know it

“…The cultural context of the AI developers and industry, e.g. their biases and prejudices, influence the way the technology is designed, developed, and implemented (Ntoutsi et al, 2020;Baker and Hawn, 2021;Pagano et al, 2023;Shachar and Gerke, 2023;Newman-Griffis et al, 2023). In turn, the AI use feeds back into reinforcing or amplifying the cultural practices that contributed to it.…”

AI thinking and the enterprise of science

“…This places assessment of AI risks, as well as AI governance, in a proactive focus on the social contexts where AI systems are designed and used, rather than a reactive, technology-centred perspective. This contextualised perspective anchors the questions of where ethical risks and failures arise and what we can do about them in specific decisions made by people and organisations during AI design and implementation, instead of after systems are extant and released into the world (11). AI thinking can therefore make management of AI risk more tractable and actionable both for internal management and external governance.…”

Putting Virtual Reality to Disability Activism

“…These programs attempt to provoke a response for the general user that helps them better "appreciate" the disability in question, although its effects are likely to be grounded in fear of the disability or a problematic sympathy for the disability, both of which can easily marginalize the personhood of people with the disability in question. The other conception we have discussed leans more toward the political/relational model of disability, which centers on disabled people's political agency (Kafer, 2013;Newman-Griffis et al, 2023).…”

“…In this article, we use the term "disabled people" based on a social model approach. Additionally, our framing is rooted in the political/relational model of disability that centers on disabled people's political agency (Kafer, 2013;Newman-Griffis et al, 2023). We supply key Japanese terms with original kanji when they are first mentioned in this article, and where it is appropriate.…”