Making the black box transparent: A template and tutorial for (pre-)registration of studies using Experience Sampling Methods (ESM)

Abstract

A growing interest in understanding complex and dynamic psychological processes as they occur in everyday life has led to an increase in studies using Ambulatory Assessment techniques, including the Experience Sampling Method (ESM) and Ecological Momentary Assessment (EMA). There are, however, numerous “forking paths” and researcher degrees of freedom, even beyond those typically encountered with other research methodologies. Whilst a number of researchers working with ESM techniques are actively engaged in efforts to increase the methodological rigor and transparency of such research, currently, there is little routine implementation of open science practices in ESM research. In the current paper, we discuss the ways in which ESM research is especially vulnerable to threats to transparency, reproducibility and replicability. We propose that greater use of (pre-)registration, a cornerstone of open science, may address some of these threats to the transparency of ESM research. (Pre-)registration of ESM research is not without challenges, including model selection, accounting for potential model convergence issues and the use of pre-existing datasets. As these may prove to be significant barriers to (pre-)registration for ESM researchers, we also discuss ways of overcoming these challenges and of documenting them in a (pre-)registration. A further challenge is that current general templates do not adequately capture the unique features of ESM. Here we present a (pre-)registration template for ESM research, adapted from the original Pre-Registration Challenge (Mellor et al., 2019) and pre-registration of pre-existing data (van den Akker et al., 2020) templates, and provide examples of how to complete this.

Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology

“…Thus, researchers need to be transparent about their choices for a reader to be able to appraise the results. Furthermore, researchers should try to mitigate data-contingent analysis decisions, for instance by pre-registration of the analysis plan, prior to observing the data ( [64]; for an ESM template see: [65]). In this project, pre-registration could have had an additional advantage: it could have made explicit that teams had different conceptualizations of the research question and therefore different analysis goals.…”

Validation of a Brief Measure of Aggression for Ecological Momentary Assessment Research: The Aggression-ES-A

“…The latter comparison is based on age 20 selfreported Aggression Social Behavior Questionnaire scores (see Measures) in the main cohort study, t(516.7) = 2.92, p = .004. The sample size of n = 255 was the largest that could be attained within resource constraints and can be expected to provide sufficient statistical power to detect key effects of interest in EMA studies (e.g., cross-level interactions, within-person correlations) assuming small to moderate effect sizes (e.g., Kirtley et al, 2019). For the current psychometric validation study; however, the most important sample size considerations were with respect to the estimation of within-and between-person loadings in the aggression measurement model, the within-person associations with external criteria, and the between-person associations with external criteria.…”

Routinely randomize potential sources of measurement reactivity to estimate and adjust for biases in subjective reports.

“…For example, while the psych R package(Revelle, 2018) implementations of multilevel generalizability(Shrout & Lane, 2012) and intraclass correlations(Shrout & Fleiss, 1979) cope well with missing data, several other packages implement intraclass correlation solutions that expect complete data.7 For further practical guidance on planning and preregistering studies including repeated measurements, see van andKirtley et al (2020). Given that van Roekel et.…”