Taxonomy of Risks posed by Language Models

Weidinger, Laura; Uesato, Jonathan; Rauh, Maribeth; Griffin, Conor; Huang, Po-Sen; Mellor, John; Glaese, Amelia; Cheng, Myra; Balle, Borja; Kasirzadeh, Atoosa; Biles, Courtney; Brown, Sasha; Kenton, Zac; Hawkins, Will; Stepleton, Tom; Birhane, Abeba; Hendricks, Lisa Anne; Rimell, Laura; Isaac, William; Haas, Julia; Legassick, Sean; Irving, Geoffrey; Gabriel, Iason

doi:10.1145/3531146.3533088

Cited by 226 publications

(188 citation statements)

References 49 publications

Supporting

Mentioning

120

Contrasting

Unclassified

Order By: Relevance

“…Another common approach is to orient harm taxonomies around specific algorithmic or model functions (e.g., [73,208]). Model-focused taxonomies have been developed for large language models [208], image captioning systems [104,202], and so-called "foundational models, " such as GPT-3 and BERT, which are applied in a wide range of downstream tasks [32]. Organizing harm by model function is highly useful when practitioners' focus is on a singular model because it draws attention to relevant computational harms.…”

Section: Taxonomies Of Sociotechnical Harms Risk and Failurementioning

confidence: 99%

“…this may undermine the profitability of creative or innovative work. " [208] 3.2.1 Opportunity loss. Opportunity loss occurs when algorithmic systems enable disparate access to information and resources needed to equitably participate in society, including withholding of housing [10] and services [74].…”

Section: Allocative Harms: Inequitable Distribution Of Resourcesmentioning

confidence: 99%

“…Diminished health and well-being can arise through purposeful behavioral exploitation [18,184] or emotional manipulation [177], whereby algorithmic designs exploit user behavior, through safety failures (e.g., collisions) [57], or when algorithmic systems make incorrect health inferences [139]. They can lead to both physical harms [61,69,120,160], emotional harms, such as distress [208], dignity loss [38,120], and misgendering [106], and reputational harms [132,160]. Diminished health and well-being may accompany other identified sociotechnical harms -for example, experiences of representational harms, including algorithmic annihilation [8] or the internalization of stereotypes, may spark other emotional or psychological effects, such as "epistemic doubt" [205], which affects overall health.…”

Section: Interpersonal Harms: Technological Affordances Adversely Sha...mentioning

confidence: 99%

“…For instance, privacy violence may arise from algorithmic systems making predictive inference beyond what users openly disclose [196] or when data collected and algorithmic inferences made about people in one context is applied to another without the person's knowledge or consent through big data flows [119], even after those datasets or systems have been deprecated [50,72]. Even if those inferences are false (e.g., the incorrect assessment of one's sexuality), people or systems can act on that information in ways that lead to discrimination and harm [208]. Privacy violations may also occur through ubiquitous surveillance, surveillance based on emotional/affective targeting [184], or coercive and exploitative data practices [100].…”

Section: Privacy Violationmentioning

confidence: 99%

See 3 more Smart Citations

Identifying Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction

Shelby¹,

Rismani²,

Henne³

et al. 2022

Preprint

View full text Add to dashboard Cite

Understanding the landscape of potential harms from algorithmic systems enables practitioners to better anticipate consequences of the systems they build. It also supports the prospect of incorporating controls to help minimize harms that emerge from the interplay of technologies and social and cultural dynamics. A growing body of scholarship has identified a wide range of harms across different algorithmic technologies. However, computing research and practitioners lack a high level and synthesized overview of harms from algorithmic systems arising at the micro-, meso-, and macro-levels of society. We present an applied taxonomy of sociotechnical harms to support more systematic surfacing of potential harms in algorithmic systems. Based on a scoping review of computing research (n=172), we identified five major themes related to sociotechnical harms -representational, allocative, quality-of-service, interpersonal harms, and social system/societal harms -and sub-themes. We describe these categories and conclude with a discussion of challenges and opportunities for future research.

show abstract

Section: Taxonomies Of Sociotechnical Harms Risk and Failurementioning

confidence: 99%

Section: Allocative Harms: Inequitable Distribution Of Resourcesmentioning

confidence: 99%

Section: Interpersonal Harms: Technological Affordances Adversely Sha...mentioning

confidence: 99%

Section: Privacy Violationmentioning

confidence: 99%

See 2 more Smart Citations

Identifying Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction

Shelby¹,

Rismani²,

Henne³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…There are a number of other ethical dimensions that have been discussed [34]. While TEAL does not assess these dimensions outof-the-box, if a user can define a function that quantifies these risks as a function of generated language, they can be assessed.…”

Section: Other Dimensionsmentioning

confidence: 99%

Democratizing Ethical Assessment of Natural Language Generation Models

Rasekh¹,

Eisenberg²

2022

Preprint

View full text Add to dashboard Cite

Natural language generation models are computer systems that generate coherent language when prompted with a sequence of words as context. Despite their ubiquity and many beneficial applications, language generation models also have the potential to inflict social harms by generating discriminatory language, hateful speech, profane content, and other harmful material. Ethical assessment of these models is therefore critical. But it is also a challenging task, requiring an expertise in several specialized domains, such as computational linguistics and social justice. While significant strides have been made by the research community in this domain, accessibility of such ethical assessments to the wider population is limited due to the high entry barriers. This article introduces a new tool to democratize and standardize ethical assessment of natural language generation models: Tool for Ethical Assessment of Language generation models (TEAL), a component of Credo AI Lens, an open-source assessment framework. CCS CONCEPTS• Computing methodologies → Natural language generation.

show abstract

Enhancing Postmarketing Surveillance of Medical Products With Large Language Models

Matheny,

Yang,

Smith

et al. 2024

JAMA Netw Open

View full text Add to dashboard Cite

ImportanceThe Sentinel System is a key component of the US Food and Drug Administration (FDA) postmarketing safety surveillance commitment and uses clinical health care data to conduct analyses to inform drug labeling and safety communications, FDA advisory committee meetings, and other regulatory decisions. However, observational data are frequently deemed insufficient for reliable evaluation of safety concerns owing to limitations in underlying data or methodology. Advances in large language models (LLMs) provide new opportunities to address some of these limitations. However, careful consideration is necessary for how and where LLMs can be effectively deployed for these purposes.ObservationsLLMs may provide new avenues to support signal-identification activities to identify novel adverse event signals from narrative text of electronic health records. These algorithms may be used to support epidemiologic investigations examining the causal relationship between exposure to a medical product and an adverse event through development of probabilistic phenotyping of health outcomes of interest and extraction of information related to important confounding factors. LLMs may perform like traditional natural language processing tools by annotating text with controlled vocabularies with additional tailored training activities. LLMs offer opportunities for enhancing information extraction from adverse event reports, medical literature, and other biomedical knowledge sources. There are several challenges that must be considered when leveraging LLMs for postmarket surveillance. Prompt engineering is needed to ensure that LLM-extracted associations are accurate and specific. LLMs require extensive infrastructure to use, which many health care systems lack, and this can impact diversity, equity, and inclusion, and result in obscuring significant adverse event patterns in some populations. LLMs are known to generate nonfactual statements, which could lead to false positive signals and downstream evaluation activities by the FDA and other entities, incurring substantial cost.Conclusions and RelevanceLLMs represent a novel paradigm that may facilitate generation of information to support medical product postmarket surveillance activities that have not been possible. However, additional work is required to ensure LLMs can be used in a fair and equitable manner, minimize false positive findings, and support the necessary rigor of signal detection needed for regulatory activities.

show abstract

Taxonomy of Risks posed by Language Models

Cited by 226 publications

References 49 publications

Identifying Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction

Identifying Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction

Democratizing Ethical Assessment of Natural Language Generation Models

Enhancing Postmarketing Surveillance of Medical Products With Large Language Models

Contact Info

Product

Resources

About