An Annotation Protocol for Collecting User-Generated Counter-Arguments Using Crowdsourcing

Abstract

Constructive feedback is important for improving critical thinking skills. However, little work has been done to automatically generate such feedback for an argument. In this work, we experiment with an annotation protocol for collecting user-generated counter-arguments via crowdsourcing. We conduct two parallel crowdsourcing experiments, where workers are instructed to produce i) a counter-argument, and ii) a counter-argument after identifying a fallacy. Our analysis indicates that we can collect counter-arguments that are useful as constructive feedback, especially when workers are first asked to identify a fallacy type.

Strategic Argumentation Dialogues for Persuasion: Framework and Experiments Based on Modelling the Beliefs and Concerns of the Persuadee

“…Generally, it can be seen as a source of knowledge for the APS, and has to have sufficient breadth and depth in order to tackle any issues that may be raised during a persuasion attempt. It can be populated manually [HH19,RII19], through argument mining [LT16,SS19], through argument harvesting/crowdsourcing [CHHP18,CHHP19], or by any combination of these methods. The contents of an argument can be textual, as seen in natural language dialogues or newspapers, or follow a particular logical structure, as obtained from formal knowledge bases.…”