Social computing researchers are beginning to apply machine learning tools to classify and analyze social media data. Our interest in understanding politeness in an online community focused our attention on tools that would help automate politeness analysis. This paper highlights one popular classification tool designed to score the politeness of text. Our application of this tool to Wikipedia data yielded some unexpected results. Those unexpected results led us to question how the tool worked and its effectiveness relative to human judgment and classification. We designed a user study to revalidate the tool with crowdworkers labeling samples of content from Wikipedia talk pages, imitating the original efforts to validate the tool. This revalidation points to challenges for automatic labeling. Based on our results, this paper reconsiders politeness in online communities as well as broader trends in the use of machine classifiers in social computing research.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.