Routinely randomize potential sources of measurement reactivity to estimate and adjust for biases in subjective reports.

Abstract

With the advent of online and app-based studies, researchers in psychology are making increasing use of repeated subjective reports. The new methods open up opportunities to study behavior in the field and to map causal processes, but they also pose new challenges. Recent work has added initial elevation bias to the list of common pitfalls; here, higher negative states (i.e., thoughts and feelings) are reported on the first day of assessment than on later days. This article showcases a new approach to addressing this and other measurement reactivity biases. Specifically, we employed a planned missingness design in a daily diary study of more than 1,300 individuals who were assessed over a period of up to 70 days to estimate and adjust for measurement reactivity biases. We found that day of first item presentation, item order, and item number were associated with only negligible bias: items were not answered differently depending on when and where they were shown. Initial elevation bias may thus be more limited than has previously been reported or it may act only at the level of the survey, not at the item level. We encourage researchers to make design choices that will allow them to routinely assess measurement reactivity biases in their studies. Specifically, we advocate the routine randomization of item display and order, as well as of the timing and frequency of measurement. Randomized planned missingness makes it possible to empirically gauge how fatigue, familiarity, and learning interact to bias responses.

A mixed-method investigation into measurement reactivity to the experience sampling method: The role of sampling protocol and individual characteristics

“…Specifically, we detected consistent decreases in the within-person variance of responses, in line with previous studies (Csikszentmihalyi & Larson, 1987;Fuller-Tyszkiewicz et al, 2013;Vachon et al, 2016). In addition, participants were becoming faster at responding to questions over time, which is also in line with previous results (Arslan, Reitz, Driebe, Gerlach, & Penke, 2020;Husky et al, 2010;Johnson et al, 2009;Labhart et al, 2020). The predicted decrease from 4.4s per item on day 1 to 3.4s per item on day 14 is comparable to what has been reported in a previous diary study, where response times evolved from 5s per item on day 1 to 2-2.5s on day 30 of data collection (Arslan et al, 2020).…”

“…In addition, participants were becoming faster at responding to questions over time, which is also in line with previous results (Arslan, Reitz, Driebe, Gerlach, & Penke, 2020;Husky et al, 2010;Johnson et al, 2009;Labhart et al, 2020). The predicted decrease from 4.4s per item on day 1 to 3.4s per item on day 14 is comparable to what has been reported in a previous diary study, where response times evolved from 5s per item on day 1 to 2-2.5s on day 30 of data collection (Arslan et al, 2020). Alongside these changes in the data, many participants reported becoming more habituated to the measures over time in interviews, as has been documented in one previous study (Paterson, Primeau, & Lauder, 2019).…”

“…The mechanism underlying this change has not been identified, meaning that it could be caused by both changes in the underlying construct, as well as by changes in response behavior. However, a recent study that investigated the initial elevation bias in diary data has not detected changes consistent with such an effect (Arslan, Reitz, Driebe, Gerlach, & Penke, 2020).…”

Mental Health and Social Contact During the COVID-19 Pandemic: An Ecological Momentary Assessment Study

“…However, recent systematic investigations have raised doubts about initial elevation bias in the context of EMA. In a daily diary study of more than 1,300 participants, Arslan et al (2020) assessed constructs similar to those included in our study (e.g., stress, loneliness, mood) and concluded that findings are likely substantive in nature rather than measurement artifacts.…”

Comparison of Two Ecological Momentary Intervention Modules for Treatment of Depression on Momentary Positive and Negative Affect

“…Ideally one would add a neutral EMA control group to disentangle the content-specific effects from more general effects of selfmonitoring. However, whether it is possible to create such a truly neutral control condition is subject of debate, since monitoring of affect can already be regarded as an interven- Arslan et al, 2020;or Bos et al, 2019). Hence, the lack of such a group is an inevitable limitation to this design.…”