Proceedings of the 10th Workshop on Argument Mining

Abstract

Previous work on the automatic identification of fallacies in natural language text has typically approached the problem in constrained experimental setups that make it difficult to understand the applicability and usefulness of the proposals in the real world. In this paper, we present the first analysis of the limitations that these data-driven approaches could show in real situations. For that purpose, we first create a validation corpus consisting of natural language argumentation schemes. Second, we provide new empirical results to the emerging task of identifying fallacies in natural language text. Third, we analyse the errors observed outside of the testing data domains considering the new validation corpus. Finally, we point out some important limitations observed in our analysis that should be taken into account in future research in this topic. Specifically, if we want to deploy these systems in the Wild. Arg. Scheme CQs Natural Language Argumentation Schemes AFEO ✓ Major Premise: "Prof Whittaker is a professor of virology at the Cornell University College" Minor Premise: "Prof Whittaker said that viruses can be spread by sneezing" Conclusion: "Viruses can be spread by sneezing" AFEO ✗ Major Premise: "Stephen Hawking was an expert on AI" Minor Premise: "Stephen Hawking said that AI could spell the end of the human race" Conclusion: "AI could spell the end of the human race" AFPK ✓ Major Premise: "Alice lives in New York" Minor Premise: "Alice says that New York City Hall is in Lower Manhattan" Conclusion: "New York City Hall is in Lower Manhattan" AFPK ✗ Major Premise: "David is a cab driver in London" Minor Premise: "David says that the best way to get to Tower Bridge is by cab" Conclusion: "The best way to get to Tower Bridge is by cab" AFPP ✓ Major Premise: "Most people wear black clothes at a funeral" Minor Premise: "If most people wear black clothes at a funeral, that is acceptable to do" Conclusion: "It is acceptable to wear black clothes at a funeral" AFPP ✗ Major Premise: "Most people drive at least 10 miles per hour over the speed limit" Minor Premise: "If most people drive at least 10 miles per hour over the speed... ...limit, that is acceptable to do" Conclusion: "It is acceptable to drive at least 10 miles per hour over the speed limit" AFPO ✓ General Acceptance Premise: "The majority of climate scientists agree that humans... ...are causing global warming and climate change" Presumption Premise: "If the majority of climate scientists agree that humans... ...are causing global warming and climate change, there is a reason to believe that is true" Conclusion: "There is reason to believe that humans... ...are causing global warming and climate change" AFPO ✗General Acceptance Premise: "The majority of people we asked agreed that the Earth may be flat " Presumption Premise: "If the majority of people we asked agreed that the Earth... ...may be flat, there is a reason to believe that is true" Conclusion: "There is reason to believe that the Earth may be flat" SSAS ✓ FirstStep Premise: "I should ...