A Total Error Framework for Digital Traces of Human Behavior on Online Platforms

Abstract

People’s activities and opinions recorded as digital traces online, especially on social media and other web-based platforms, offer increasingly informative pictures of the public. They promise to allow inferences about populations beyond the users of the platforms on which the traces are recorded, representing real potential for the social sciences and a complement to survey-based research. But the use of digital traces brings its own complexities and new error sources to the research enterprise. Recently, researchers have begun to discuss the errors that can occur when digital traces are used to learn about humans and social phenomena. This article synthesizes this discussion and proposes a systematic way to categorize potential errors, inspired by the Total Survey Error (TSE) framework developed for survey methodology. We introduce a conceptual framework to diagnose, understand, and document errors that may occur in studies based on such digital traces. While there are clear parallels to the well-known error sources in the TSE framework, the new “Total Error Framework for Digital Traces of Human Behavior on Online Platforms” (TED-On) identifies several types of error that are specific to the use of digital traces. By providing a standard vocabulary to describe these errors, the proposed framework is intended to advance communication and research about using digital traces in scientific social research.

Realizing a Global Survey of Emigrants through Facebook and Instagram

“…Most important in this regard are its non-probability character and the associated selection biases. Advertisements on networks such as Facebook and Instagram can obviously only reach, in a direct way, individuals who use these SNS (Sen et al 2021). On top of this, there might be a self-selection bias, meaning that those target group users who react to such advertisements and eventually participate in the survey might differ systematically from those who do not.…”

“I updated the <ref>”: The evolution of references in the English Wikipedia and the implications for altmetrics

“…The situation quickly improved between 2009 and 2010 (11.3%), and then continued doing so until our data collection. Our findings show that mining methods that rely on DIDs are vulnerable to coverage errors (Sen et al, 2021) that can misrepresent the importance of academic works in the altmetrics community.…”

“…We assume that these wide differences are not only due to the diverse sets of publications covered by the aggregators but also due to their distinct methods of tracing Wikipedia references that are prone to various errors considering the challenges inherent to Wikipedia data. Besides the difficulties of keeping track of continuous changes in Wikipedia where references may be modified or removed, one important source of coverage errors (Sen et al, 2021) is the reliance on standard document identifiers to trace publications (Ortega, 2018). Similarly, other approaches that rely on explicit bibliographic information, such as title and first author name (Kousha & Thelwall, 2017) fail to identify references that do not specify this information in the provided fields (Pooladian & Borrego, 2017).…”

“…To the best of our knowledge, there are no studies that investigate in detail how common self-citations are in Wikipedia or that aim to identify misconduct in the area of self-promoting scientific articles through Wikipedia. 14 One example can be found at: https://web.archive.org/web/20200323131800/https://annualreviewsnews.org/2020/02/25/seeking-a-wikipedian-inresidence/ Zagovora, O., Ulloa, R., Weller, K., Flöck, F. (2021) 'I Updated the <ref>': The Evolution of References in the English Wikipedia and the Implications for Altmetrics. Quantitative Science Studies.…”

“…The first column presents the cluster id and the percentage of registered editors in parenthesis. The second column describes the group of registered editors in terms of the actions they perform (FigureL1aof Appendix L in the Supplementary material).Zagovora, O., Ulloa, R., Weller, K.,Flöck, F. (2021) 'I Updated the <ref>': The Evolution of References in the English Wikipedia and the Implications for Altmetrics. Quantitative Science Studies.…”