Longitudinal Assessment of Reference Quality on Wikipedia

Abstract

Wikipedia plays a crucial role in the integrity of the Web. This work analyzes the reliability of this global encyclopedia through the lens of its references. We operationalize the notion of reference quality by defining reference need (RN), i.e., the percentage of sentences missing a citation, and reference risk (RR), i.e., the proportion of non-authoritative references. We release Citation Detective, a tool for automatically calculating the RN score, and discover that the RN score has dropped by 20 percent point in the last decade, with more than half of verifiable statements now accompanying references. The RR score has remained below 1% over the years as a result of the efforts of the community to eliminate unreliable references. We propose pairing novice and experienced editors on the same Wikipedia article as a strategy to enhance reference quality. Our quasi-experiment indicates that such a co-editing experience can result in a lasting advantage in identifying unreliable sources in future edits. As Wikipedia is frequently used as the ground truth for numerous Web applications, our findings and suggestions on its reliability can have a far-reaching impact. We discuss the possibility of other Web services adopting Wiki-style user collaboration to eliminate unreliable content. CCS CONCEPTS• Information systems → Wikis.

Disappearing without a Trace: Coverage, Community, Quality, and Temporal Dynamics of Wikipedia Articles on Endangered Brazilian Indigenous Languages

“…Given its popularity and accessibility, many researchers have conducted studies in Wikipedia, exploring various aspects of the platform, such as its content, structure, and social dynamics. Those studies have provided insights into issues such as automatically assessing article quality (Dalip et al 2009;Shen, Qi, and Baldwin 2017), analyzing citations (Piccardi et al 2020;Baigutanova et al 2023), images (He et al 2018;Rama et al 2022), andinfo-boxes (Graells-Garrido, Lalmas, andMenczer 2015;Lewoniewski 2017) or understanding readers' preferences (Lehmann et al 2014). Other works focused on identifying underrepresented groups (Graells-Garrido, Lalmas, and Menczer 2015;Mandiberg 2023;Gallert et al 2016;Sethuraman, Grinter, and Zegura 2020;Hoenen and Rahn 2021) and the asymmetry of the coverage across different language versions (Hale 2015;Graham 2011;Lemmerich et al 2019;Roy, Bhatia, and Jain 2020;Ashrafimoghari 2023).…”

“…We analyze the citations listed in the References section of each Wikipedia article, including citations with and without links to external sources. Citations are crucial, as Wikipedia guidelines require editors to support their edits with appropriate references to ensure verifiability (Baigutanova et al 2023). Indeed, the number of citations (ref wo link) and links (ref w link) in the Reference sections correlates with the ORES metric, as show in Figure 6.…”