Case Study: Deontological Ethics in NLP

Prabhumoye, Shrimai; Boldt, Brendon; Salakhutdinov, Ruslan; Black, Alan W.

doi:10.18653/v1/2021.naacl-main.297

Cited by 16 publications

(11 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…And since morally informed NLP systems perpetuate patterns that are present in their training data (i.e., data on huge troves of moral judgments made by people), they represent a descriptive approach to ethics. These morally informed AI systems do not derive their reports from a particular ethical theory's framework or moral axioms in a prescriptive manner [13], but instead reflect empirically observed patterns of judgments. Whenever this approach fails to encode the "right" patterns (as assessed by the AI system developers), prescriptive approaches are harnessed to correct crowdsourced data.…”

Section: Morally Informed Ai Systemsmentioning

confidence: 99%

Ethical and methodological challenges in building morally informed AI systems

Hagendorff

Danks

2022

AI Ethics

View full text Add to dashboard Cite

Recent progress in large language models has led to applications that can (at least) simulate possession of full moral agency due to their capacity to report context-sensitive moral assessments in open-domain conversations. However, automating moral decision-making faces several methodological as well as ethical challenges. They arise in the fields of bias mitigation, missing ground truth for moral “correctness”, effects of bounded ethicality in machines, changes in moral norms over time, risks of using morally informed AI systems as actual advice, as well as societal implications an increasing importance of algorithmic moral decision-making would have. This paper comments on all these challenges and provides critical considerations for future research on full artificial moral agency. Importantly, some of the adduced challenges can be met by more careful technology design, but others necessarily require engagement with core problems of meta-ethics.

show abstract

Section: Morally Informed Ai Systemsmentioning

confidence: 99%

Ethical and methodological challenges in building morally informed AI systems

Hagendorff

Danks

2022

AI Ethics

View full text Add to dashboard Cite

show abstract

“…There is a long-standing interest in the moral responsibility of AI (Dehghani et al, 2008;Alaieri and Vellino, 2016;Stephanidis et al, 2019;Zoshak and Dew, 2021;Prabhumoye et al, 2021;Schramowski et al, 2021). Work in Human-Computer Interaction (HCI) reveals that, before users feel they can trust a Conversational Agent, they will often probe it to identify the limitations which bound its abilities, competence (Luger and Sellen, 2016), and apparent integrity (Mayer et al, 1995;McKnight et al, 2002;Wang and Benbasat, 2016).…”

Section: Related Workmentioning

confidence: 99%

The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems

Ziems¹,

Yu²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Content Warning: some examples in this paper may be offensive or upsetting.Conversational agents have come increasingly closer to human competence in open-domain dialogue settings; however, such models can reflect insensitive, hurtful, or entirely incoherent viewpoints that erode a user's trust in the moral integrity of the system. Moral deviations are difficult to mitigate because moral judgments are not universal, and there may be multiple competing judgments that apply to a situation simultaneously. In this work, we introduce a new resource, not to authoritatively resolve moral ambiguities, but instead to facilitate systematic understanding of the intuitions, values and moral judgments reflected in the utterances of dialogue systems. The MORAL INTEGRITY CORPUS, MIC , is such a resource, which captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs). Each RoT reflects a particular moral conviction that can explain why a chatbot's reply may appear acceptable or problematic. We further organize RoTs with a set of 9 moral and social attributes and benchmark performance for attribute classification. Most importantly, we show that current neural language models can automatically generate new RoTs that reasonably describe previously unseen interactions, but they still struggle with certain scenarios. Our findings suggest that MIC will be a useful resource for understanding and language models' implicit moral assumptions and flexibly benchmarking the integrity of conversational agents. To download the data, see https://github.com/GT-SALT/mic

show abstract

“…Hendrycks et al (2021) argue that works on fairness, safety, prosocial behavior, and utility of machine learning systems in fact address parts of broader theories in normative ethics, such as the concept of justice, deontological ethics, virtue ethics, and utilitarianism. Card and Smith (2020) and Prabhumoye et al (2021) show how NLP research and applications can be grounded in established ethical theories. Ziems et al (2022) presents a corpus annotated for moral "rules-of-thumb" to help explain why a chatbot's reply may be considered problematic under various moral assumptions.…”

Section: Ethics In Machine Learning and Nlpmentioning

confidence: 99%

Does Moral Code Have a Moral Code? Probing Delphi's Moral Philosophy

Fraser¹,

Kiritchenko²,

Balkir³

2022

Preprint

View full text Add to dashboard Cite

In an effort to guarantee that machine learning model outputs conform with human moral values, recent work has begun exploring the possibility of explicitly training models to learn the difference between right and wrong. This is typically done in a bottom-up fashion, by exposing the model to different scenarios, annotated with human moral judgements. One question, however, is whether the trained models actually learn any consistent, higher-level ethical principles from these datasets -and if so, what? Here, we probe the Allen AI Delphi model with a set of standardized morality questionnaires, and find that, despite some inconsistencies, Delphi tends to mirror the moral principles associated with the demographic groups involved in the annotation process. We question whether this is desirable and discuss how we might move forward with this knowledge.

show abstract

Case Study: Deontological Ethics in NLP

Cited by 16 publications

References 79 publications

Ethical and methodological challenges in building morally informed AI systems

Ethical and methodological challenges in building morally informed AI systems

The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems

Does Moral Code Have a Moral Code? Probing Delphi's Moral Philosophy

Contact Info

Product

Resources

About