Wiki-Reliability: A Large Scale Dataset for Content Reliability on Wikipedia

Abstract

Wikipedia is the largest online encyclopedia, used by algorithms and web users as a central hub of reliable information on the web. The quality and reliability of Wikipedia content is maintained by a community of volunteer editors. Machine learning and information retrieval algorithms could help scale up editors' manual efforts around Wikipedia content reliability. However, there is a lack of large-scale data to support the development of such research. To fill this gap, in this paper, we propose Wiki-Reliability, the first dataset of English Wikipedia articles annotated with a wide set of content reliability issues.To build this dataset, we rely on Wikipedia "templates". Templates are tags used by expert Wikipedia editors to indicate content issues, such as the presence of "non-neutral point of view" or "contradictory articles", and serve as a strong signal for detecting reliability issues in a revision. We select the 10 most popular reliability-related templates on Wikipedia, and propose an effective method to label almost 1M samples of Wikipedia article revisions as positive or negative with respect to each template. Each positive/negative example in the dataset comes with the full article text and 20 features from the revision's metadata. We provide an overview of the possible downstream tasks enabled by such data, and show that Wiki-Reliability can be used to train large-scale models for content reliability prediction. We release all data and code for public use.

WikiSQE: A Large-Scale Dataset for Sentence Quality Estimation in Wikipedia

“…Previous studies on quality estimation in Wikipedia have mainly focused on the article-level (Mola-Velasco 2011; Bykau et al 2015;Wong, Redi, and Saez-Trumper 2021;Asthana et al 2021). They are aimed at estimating the quality of revisions and articles.…”

Interpretable Classification of Wiki-Review Streams

“…According to the literature, the classification of wiki edits encompasses the detection of paid [20], puffery [21], reverted [22], [23], [24], toxic [25], [26] and vandal [9], [10], [12], [13], [17], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38] reviews. Similarly, prediction focuses on review quality [39], [40], [41] as well as on editor and article quality [18], [42], [43], [44], [45].…”

“…In the case of article drafts, ORES returns the probability of being spam, vandalism, an attack, and OK. These scores are used as input features by many of the surveyed works, e.g., [9], [34], [36], [37], [41], to classify reviews. Moreover, ORES is currently used on wiki platforms to help volunteers reduce the burden of manually screening content.…”

“…Using side-based and stylometric profiles, the solution applies a deep neural network to extract quality indicators. [41] shared an annotated data set of English Wikipedia articles based on Wikipedia templates, e.g., original research, contradictory, unreliable sources, etc. The data set was used to predict the content reliability using Logistic Regression [49], Random Forest, and Gradient Boosted Trees [50].…”