Predicting quality flaws in user-generated content

Abstract

The detection and improvement of low-quality information is a key concern in Web applications that are based on user-generated content; a popular example is the online encyclopedia Wikipedia. Existing research on quality assessment of user-generated content deals with the classification as to whether the content is high-quality or low-quality. This paper goes one step further: it targets the prediction of quality flaws, this way providing specific indications in which respects low-quality content needs improvement. The prediction is based on user-defined cleanup tags, which are commonly used in many Web applications to tag content that has some shortcomings. We apply this approach to the English Wikipedia, which is the largest and most popular user-generated knowledge source on the Web. We present an automatic mining approach to identify the existing cleanup tags, which provides us with a training corpus of labeled Wikipedia articles. We argue that common binary or multiclass classification approaches are ineffective for the prediction of quality flaws and hence cast quality flaw prediction as a one-class classification problem. We develop a quality flaw model and employ a dedicated machine learning approach to predict Wikipedia's most important quality flaws. Since in the Wikipedia setting the acquisition of significant test data is intricate, we analyze the effects of a biased sample selection. In this regard we illustrate the classifier effectiveness as a function of the flaw distribution in order to cope with the unknown (real-world) flawspecific class imbalances. The flaw prediction performance is evaluated with 10 000 Wikipedia articles that have been tagged with the ten most frequent quality flaws: provided test data with little noise, four flaws can be detected with a precision close to 1.

Cluster-based one-class ensemble for classification problems in information retrieval

“…The parameters of classifier cj are optimized on the respective training set Cj ⊆ Dtrain ; clusters with less than five elements are discarded. Figure 1 illustrates the results on four different datasets: artificially created objects with three clusters (plot in Figure 2), documents from the 20 Newsgroups dataset with category "computer" in the role of the target class, books from different authors for which the authorship is to be verified [4], and Wikipedia articles tagged with certain quality flaws that are to be detected [1]. All documents are represented under a vector space model with a tf-idf weighting except for Wikipedia articles where quality-specific features [1] are employed.…”

Relative Quality and Popularity Evaluation of Multilingual Wikipedia Articles

“…Before transferring values of particular parameters of infobox, information are compared to other language versions, but versions with higher quality and popularity scores will have higher influence (weight) on selecting the proper value. The methods proposed in the paper are used in WikiRank.net service 10 , which assesses and compare articles in the various language versions of Wikipedia.…”

“…Basic lexical metrics based on word usages in Wikipedia articles used in another study as the factors that can reflect articles quality -high-quality articles often used more nouns and verbs and less adjectives [9]. Finally, quality evaluation of Wikipedia articles can also base on special quality flaw templates [10].…”

“…Such template in English Wikipedia can be divided into 12 types, e.g. verifiability, style of writing, structure, neutrality and others [10]. We conducted a preliminary analysis of the best articles for finding quality flaw templates.…”

Insights into Entity Name Evolution on Wikipedia

“…It denotes the task of automatically detecting flaws according to Wikipedia's guidelines, something not to neglect when working with Wikipedia. Anderka et al [8] have done an impressive work in this field and give a nice overview of the first challenge dedicated to this topic [9]. Another related topic is the research on Wikipedia's revision history and talk pages.…”