Applying multiverse analysis to experience sampling data: Investigating whether preprocessing choices affect robustness of conclusions

Abstract

The experience sampling method (ESM) has revolutionized our ability to conduct psychological research in the natural environment. However, researchers have a large degree of freedom when preprocessing ESM data, which may hinder scientific progress. This study illustrates the use of multiverse analyses regarding preprocessing choices related to data exclusion (i.e., based on various levels of compliance and exclusion of the first assessment day) and the calculation of constructs (i.e., composite scores calculated as the mean, median, or mode) by reanalyzing established group differences in negative affect, stress reactivity, and emotional inertia between individuals with and without psychosis. Data came from five studies and included 233 individuals with psychosis and 223 healthy individuals (in total, 26,892 longitudinal assessments). Preprocessing choices related to data exclusion did not affect conclusions. For both stress reactivity and emotional inertia of negative affect, group differences were affected when negative affect was calculated as the mean compared to the median or mode. Further analyses revealed that this could be attributed to considerable differences in the within-and between-factor structure of negative affect. While these findings show that observed differences in affective processes between individuals with and without psychosis are robust to preprocessing choices related to data exclusion, we found disagreement in conclusions between different central tendency measures. Safeguarding the validity of future experience sampling research, scholars are advised to use multiverse analysis to evaluate the robustness of their conclusions across different preprocessing scenarios.

A mixed-method investigation into measurement reactivity to the experience sampling method: The role of sampling protocol and individual characteristics.

“…To further our understanding of the underlying mechanisms once robust effects are established, carefully designed experiments (along the lines of De Vuyst et al, 2019; Johar & Sackett, 2018; Shrout et al, 2018) or the analysis of data sets that combine both objective and self-report measures (Verbeij et al, 2021) seem promising approaches. Relatedly, more work is needed to judge the practical significance of the observed affects, for example, by investigating how reactive changes affect results of analyses on a practical level (see, e.g., Weermeijer et al, 2022). Furthermore, it is difficult to compare the size of observed effects to other effects in the ESM literature due to the large variability in ESM study designs and the absence of an agreement on standardized effect sizes for multilevel model (Rights & Sterba, 2019).…”

Generalizability Crisis Meets Heterogeneity Revolution: Determining Under Which Boundary Conditions Findings Replicate and Generalize

“…Otherwise one explored branch may appear plausible while three unexplored paths may have been more insightful (Gelman & Loken, 2014). Multiverse analyses and many-analysts studies, both examining heterogeneity in data analyses, can help overcome forking (Aczel et al, 2021;Bastiaansen et al, 2020;Dragicevic et al, 2019;Silberzahn et al, 2018;Steegen et al, 2016;Weermeijer et al, 2022).…”

Excess death estimates from multiverse analysis in 2009–2021

“…When results depend on analytical choices, one methodological strategy is to explore the full range of results that can be obtained when a wide range of possible analytical choices and combinations thereof are considered [11][12][13][14][15][16][17][18][19][20].…”

“…Different terminology has been used for such approaches that generalize the concept of sensitivity analysis. Commonly used terms are "multiverse analysis" [11][12][13][14], "vibration of effects" [16][17][18] and "multi-analyst analysis" [19,20] (when multiple researchers are each asked to select independently their preferred analysis). Here, we propose a multiverse approach for excess deaths.…”