StRE: Self Attentive Edit Quality Prediction in Wikipedia

Abstract

Wikipedia can easily be justified as a behemoth, considering the sheer volume of content that is added or removed every minute to its several projects. This creates an immense scope, in the field of natural language processing toward developing automated tools for content moderation and review. In this paper we propose Self Attentive Revision Encoder (StRE) which leverages orthographic similarity of lexical units toward predicting the quality of new edits. In contrast to existing propositions which primarily employ features like page reputation, editor activity or rule based heuristics, we utilize the textual content of the edits which, we believe contains superior signatures of their quality. More specifically, we deploy deep encoders to generate representations of the edits from its text content, which we then leverage to infer quality. We further contribute a novel dataset containing ∼ 21M revisions across 32K Wikipedia pages and demonstrate that StRE outperforms existing methods by a significant margin -at least 17% and at most 103%. Our pretrained model achieves such result after retraining on a set as small as 20% of the edits in a wikipage. This, to the best of our knowledge, is also the first attempt towards employing deep language models to the enormous domain of automated content moderation and review in Wikipedia. 1 en.wikipedia.org/Wikipedia:List of policies 2 stats.wikimedia.org/EN/PlotsPngEditHistoryTop.htm

Mining the online infosphere: A survey

“…A complementary direction of exploration has been put forward by [84,31] where correlation between article quality and structural properties of co-editor network and editor-article network has been exploited. An orthogonal direction of research looks into edit level quality prediction which is a fine-grained approach toward article content management [120].…”

“…We shall primarily discuss two of the most important aspects of Wikipedia -(a) the quality of an article and its indicators and (b) the collaboration dynamics of Wikipedia editors who constitute the backbone of this massive initiative. Under the first topic we shall identify the different features of an article like its language, structure and stability as well as their quality [150,53,120]. We shall further summarise attempts that have been made to automatically predict the quality of an article [52,91].…”

ArgRewrite V.2: an annotated argumentative revisions corpus

“…Most related to our work is the Wikipedia revision analysis and categorization (Daxenberger and Gurevych, 2013;Bronner and Monz, 2013;Sarkar et al, 2019). Revision categorization of user edits from Wikipedia focus on both coarse-level (Bronner and Monz, 2013) and fine-grained (Daxenberger and Gurevych, 2012;Yang et al, 2017;Jones, 2008) categories.…”

Mining the online infosphere: A survey

“…A complementary direction of exploration has been put forward by de La Robertie et al (2015) and Li, Tang, et al (2015) where the correlation between article quality and structural properties of co‐editor network and editor‐article network has been exploited. An orthogonal direction of research looks into edit level quality prediction, which is a fine‐grained approach toward article content management (Sarkar et al, 2019).…”

“…We shall primarily discuss two of the most important aspects of Wikipedia—(a) the quality of an article and its indicators and (b) the collaboration dynamics of Wikipedia editors who constitute the backbone of this massive initiative. Under the first topic, we shall identify the different features of an article like its language, structure, and stability as well as their quality (Halfaker & Geiger, 2020; Sarkar et al, 2019; Zhang et al, 2020). We shall further summarize attempts that have been made to automatically predict the quality of an article (Guda et al, 2020; Marrese‐Taylor et al, 2019).…”