Two Half-Truths Make a Whole? On Bias in Self-Reports and Tracking Data

Abstract

The pervasive use of mobile information technologies brings new patterns of media usage, but also challenges to the measurement of media exposure. Researchers wishing to, for example, understand the nature of selective exposure on algorithmically driven platforms need to precisely attribute individuals' exposure to specific content. Prior research has used tracking data to show that survey-based self-reports of media exposure are critically unreliable. So far, however, little effort has been invested into assessing the specific biases of tracking methods themselves. Using data from a multi-method study, we show that tracking data from mobile devices is linked to systematic distortions in self-report biases. Further inherent but unobservable sources of bias, along with potential solutions, are discussed.The paper is structured as follows: We first review the existing state of research on the relation between self-reports and tracking data. Building on that literature, we discuss the theoretical and pragmatic limitations in the data collection process of various tracking methods, with a special focus on mobile devices.As that section will reveal, there are numerous potential sources of errors at various stages in the data collection process, which warrant an investigation into biases in tracking data. We go on to show empirically that such biases exist, drawing on original data from a multi-method study comprising survey and tracking data. In order to establish the validity of data and method, we first replicate existing findings of biased selfassessments (RQ 1). Using the differences between participants who provided mobile and/or desktop tracking data, we then show a genuinely new type of bias, namely a differential bias in self-reports of people willing to share mobile tracking data (RQ 2). Finally, we assess the impact of this bias through a simulation exercise (RQ 3), which builds on a realistic statistical model of perceived polarization to show how strong tracking bias will impact results. Literature Review: Self-report Bias, Direction and SourcesA growing list of study designs aims to bypass the insufficient reliability of self-reports by directly capturing trace data of digital media usage through various means (Revilla et al. 2017, Araujo et al. 2017, Vraga et al. 2016, Scharkow 2016. The results show rather consistently that there are strong systematic biases present in self-reports across different devices, settings and operationalizations. An early study that served to draw attention to the issues is Prior's (2009b) investigation of time spent on TV. By comparing survey-based self-assessments to Nielsen people meter data (which are generated from custom tracking devices on TVs, see Napoli 2003), he shows that individuals on average overestimate their TV usage by a factor of three, with younger respondents doing worse. Tapping into an earlier debate in political communication (Price & Zaller 1993), the paper suggests either using alternative methods for measuring exposure or instead focusing on deeper l...

Mobile data donations: Assessing self-report accuracy and sample biases with the iOS Screen Time function

“…In line with previous research on Internet (Araujo et al, 2017;Jürgens et al, 2019) and mobile phone usage (Vanden Abeele et al, 2013), we find that self-reports of smartphone usage differs strongly from the retrieved log data. People mostly underreport the duration of usage and frequency of incidents such as checking their phone or receiving push messages.…”

A Systematic Review and Meta-Analysis of Discrepancies Between Logged and Self- Reported Digital Media Use

“…While at face-value tracking methods provide accurate and valid measurements of media use, the possibility of biases and inaccuracies cannot be ignored (Scharkow, 2016). Irrespective of validity concerns, automatically logged data is not a panacea for media use measurement (Jürgens et al, 2019; Ryding & Kuss, 2020). A particular challenge for widespread adoption of such methods relates to sampling biases occurring due to differential willingness and abilities to provide such data.…”

“…Moreover, in addition to technical incompatibilities (device or system restrictions, gaps in coverage), participant biases (reactivity), and increased resource demands (time, cost, and participant burden), there are substantial ethical, security and privacy related challenges associated with tracking media use (Jürgens et al, 2019;Scharkow, 2016). Such biases and challenges, while not necessarily affecting the accuracy of the data produced, nonetheless, merit careful consideration.…”

Computational observation: Challenges and opportunities of automated observation within algorithmically curated media environments using a browser plug-in

“…Furthermore, Facebook, in particular, is commonly accessed through smartphone browsers and proprietary apps; as of today, smartphone browsers do not support the easy use of plug-ins, and proprietary apps are not observable with appropriate means. This is especially critical, as actual tracking data from mobile devices has recently revealed even higher overreporting from mobile users than from other study participants (Jürgens, Stark, & Magin, 2019).…”