The effects of assessment intensity on participant burden, compliance, within-person variance, and within-person relationships in ambulatory assessment

Abstract

Considering the very large number of studies that have applied ambulatory assessment (AA) in the last decade across diverse fields of research, knowledge about the effects that these design choices have on participants’ perceived burden, data quantity (i.e., compliance with the AA protocol), and data quality (e.g., within-person relationships between time-varying variables) is surprisingly restricted. The aim of the current research was to experimentally manipulate aspects of an AA study’s assessment intensity—sampling frequency (Study 1) and questionnaire length (Study 2)—and to investigate their impact on perceived burden, compliance, within-person variability, and within-person relationships between time-varying variables. In Study 1, students (n = 313) received either 3 or 9 questionnaires per day for the first 7 days of the study. In Study 2, students (n = 282) received either a 33- or 82-item questionnaire three times a day for 14 days. Within-person variability and within-person relationships were investigated with respect to momentary pleasant-unpleasant mood and state extraversion. The results of Study 1 showed that a higher sampling frequency increased perceived burden but did not affect the other aspects we investigated. In Study 2, longer questionnaire length did not affect perceived burden or compliance but yielded a smaller degree of within-person variability in momentary mood (but not in state extraversion) and a smaller within-person relationship between state extraversion and mood. Differences between Studies 1 and 2 with respect to the type of manipulation of assessment intensity are discussed.

Compliance in Ambulatory Assessment Studies: Investigating Study and Sample Characteristics as Predictors

“…The other indicators of participant burden (duration of measurement occasions, number of measurement occasions per day) were not associated with the compliance. Consistent with those results, experimental AA studies ( Eisele et al, 2020 ; Hasselhorn et al, 2021 ) have shown that the sampling frequency (i.e., the number of measurement occasions per day) had no effect on the compliance. Regarding the length of the AA questionnaires, previous results were mixed ( Eisele et al., 2020 ; Hasselhorn et al, 2021 ), suggesting that the processes under which possible effects occur, should be further investigated in future research.…”

“…Consistent with those results, experimental AA studies ( Eisele et al, 2020 ; Hasselhorn et al, 2021 ) have shown that the sampling frequency (i.e., the number of measurement occasions per day) had no effect on the compliance. Regarding the length of the AA questionnaires, previous results were mixed ( Eisele et al., 2020 ; Hasselhorn et al, 2021 ), suggesting that the processes under which possible effects occur, should be further investigated in future research. It might be that participants of AA studies cope differently with the diverse design characteristics that should enhance or reduce participant burden ( Hasselhorn et al, 2021 ).…”

Compliance trends in a 14-week ecological momentary assessment study of undergraduate alcohol drinkers

“…We dispute the conclusion from other work that survey length or depth doesn't affect protocol compliance. Although meta-analyses and experimental data preceding our work failed to find any between-study differences in compliance related to surveys with more questions (Ottenstein & Werner, 2021;Hasselhorn et al, 2021;Williams et al, 2021), evidence suggests that perceived burden is higher and data quality may be poorer (Eisele et al, 2022;Hasselhorn et al, 2021). Our midday survey cognitive task was distinct from our other surveys and achieved much poorer compliance rates for all but the most engaged super respondents.…”

“…Meta-analyses show that on average, participants in EMA studies respond to 75%-82% of the prompts they receive over the course of their study involvement (Jones et al, 2019;Ottenstein & Werner, 2021;Williams et al, 2021;Wrzus et al, 2022). Studies with fewer prompts per day and shorter surveys generally did not have better compliance rates than studies with more prompts per day and longer surveys, consistent with experimental evidence (Eisele et al, 2022;Hasselhorn et al, 2021). Other design factors showing no effect on compliance included type of device (e.g., personal smartphone vs. study-loaned device), whether prompts were at fixed versus random times of day, and whether studies included event-contingent responding.…”

Ecological Momentary Assessment: A Meta-Analysis on Designs, Samples, and Compliance across Research Fields

“…Unfortunately, some characteristics could not be examined (e.g., socioeconomic status of participants) because the original studies provided too little information (see also Trull & Ebner-Priemer, 2020). Further studies and meta-analyses on specific populations additionally examined effects of time-of-day and questionnaire length: The studies observed higher compliance in the evenings (Courvoisier et al, 2012;Silvia et al, 2013;van Berkel et al, 2019), while the number of questions in EMA studies hardly predicted differences in compliance (Hasselhorn et al, 2021;Jones et al, 2019;van Roekel, 2019van Roekel, et al, 2019, but see Eisele et al, 2020;Morren et al, 2009)-likely because most EMA studies already apply short questionnaires.…”