The Questionable Ecological Validity of Ecological Momentary Assessment: Considerations for Design and Analysis

Ram, Nilam; Brinberg, Miriam; Pincus, Aaron L.; Conroy, David E.

doi:10.1080/15427609.2017.1340052

Cited by 95 publications

(76 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Focusing on group averages is common in the field of wellbeing (and beyond), but this leads to missing important information about individual differences. In EMA research, ignoring individual differences also reduces the ecological validity (Ram et al 2017). As individuals differ in their patterns of mood, behavior, and other experiences in life, the average person or pattern of those variables across participants does not exist in the real world.…”

Section: Analysesmentioning

confidence: 99%

Smartphone-Based Ecological Momentary Assessment of Well-Being: A Systematic Review and Recommendations for Future Studies

2020

View full text Add to dashboard Cite

Feelings of well-being and happiness fluctuate over time and contexts. Ecological Momentary Assessment (EMA) studies can capture fluctuations in momentary behavior, and experiences by assessing these multiple times per day. Traditionally, EMA was performed using pen and paper. Recently, due to technological advances EMA studies can be conducted more easily with smartphones, a device ubiquitous in our society. The goal of this review was to evaluate the literature on smartphone-based EMA in well-being research in healthy subjects. The systematic review was conducted according to the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines. Searching PubMed and Web of Science, we identified 53 studies using smartphone-based EMA of well-being. Studies were heterogeneous in designs, context, and measures. The average study duration was 12.8 days, with well-being assessed 2–12 times per day. Half of the studies included objective data (e.g. location). Only 47.2% reported compliance, indicating a mean of 71.6%. Well-being fluctuated daily and weekly, with higher well-being in evenings and weekends. These fluctuations disappeared when location and activity were accounted for. On average, being in nature and physical activity relates to higher well-being. Working relates to lower well-being, but workplace and company do influence well-being. The important advantages of using smartphones instead of other devices to collect EMAs are the easier data collection and flexible designs. Smartphone-based EMA reach far larger maximum sample sizes and more easily add objective data to their designs than palm-top/PDA studies. Smartphone-based EMA research is feasible to gain insight in well-being fluctuations and its determinants and offers the opportunity for parallel objective data collection. Most studies currently focus on group comparisons, while studies on individual differences in well-being patterns and fluctuations are lacking. We provide recommendations for future smartphone-based EMA research regarding measures, objective data and analyses.

show abstract

Section: Analysesmentioning

confidence: 99%

Smartphone-Based Ecological Momentary Assessment of Well-Being: A Systematic Review and Recommendations for Future Studies

2020

View full text Add to dashboard Cite

show abstract

“…Noncompliance is a potential threat for EMA methodologies [30,31]. Therefore, well-considered design choices have to be made to achieve sufficient data richness for the questions under study without over-burdening the participants [23,32].…”

Section: Introductionmentioning

confidence: 99%

Content validity and methodological considerations in ecological momentary assessment studies on physical activity and sedentary behaviour: a systematic review

Degroote

DeSmet

Bourdeaudhuij

et al. 2020

Int J Behav Nutr Phys Act

View full text Add to dashboard Cite

Background: Ecological momentary assessment (EMA) is a method of collecting real-time data based on repeated measures and observations that take place in participant's daily environment. EMA has many advantages over more traditional, retrospective questionnaires. However, EMA faces some challenges to reach its full potential. The aims of this systematic review are to (1) investigate whether and how content validity of the items (i.e. the specific questions that are part of a larger EMA questionnaire) used in EMA studies on physical activity and sedentary behaviour was assessed, and (2) provide an overview of important methodological considerations of EMA in measuring physical activity and sedentary behaviour. Methods: Thirty papers (twenty unique studies) were systematically reviewed and variables were coded and analysed within the following 4 domains: (1) Content validity, (2) Sampling approach, (3) Data input modalities and (4) Degree of EMA completion. Results: Only about half of the studies reported the specific items (n = 12) and the source of the items (n = 11). None of the studies specifically assessed the content validity of the items used. Only a minority (n = 5) of the studies reported any training, and one tested the comprehensibility of the EMA items. A wide variability was found in the design and methodology of the EMA. A minority of the studies (n = 7) reported a rationale for the used prompt frequency, time selection, and monitoring period. Retrospective assessment periods varied from 'now' to 'in the last 3.5 hours'. In some studies there was a possibility to delay (n = 6) or deactivate (n = 10) the prompt, and some provided reminders after the first prompt (n = 9). Conclusions: Almost no EMA studies reported the content validation of the items used. We recommend using the COSMIN checklist (COnsensus-based Standards for the selection of health Measurement INstruments) to report on the content validity of EMA items. Furthermore, as often no rationale was provided for several methodological decisions, the following three recommendations are made. First, provide a rationale for choosing the sampling modalities. Second, to ensure assessment 'in the moment', think carefully about the retrospective assessment period, reminders, and deactivation of the prompt. Third, as high completion rates are important for representativeness of the data and generalizability of the findings, report completion rates.

show abstract

“…A more fundamental limitation of prior studies on symptom relationships in BPD is that each is restricted to interindividual (i.e., between-person), groupwise analyses. However, structures uncovered from between-person analyses are conceptually (Borsboom, Mellenbergh, & van Heerden, 2003;Molenaar, 2004;Roche, Pincus, Rebar, Conroy, & Ram, 2014) and empirically (Beckmann, Wood, & Minbashian, 2010;Dowgwillo et al, 2019;Ram, Brinberg, Pincus, & Conroy, 2017;Roche, Pincus, Hyde, Conroy, & Ram, 2013;Yang et al, 2018) distinct from processes uncovered from within-subjects analyses. If theories of symptom NOMOTHETIC AND IDIOGRAPHIC PATTERNS IN BPD 6 interrelationships are meant to apply within subjects, cross-sectional analyses cannot test them directly.…”

mentioning

confidence: 99%

Dynamics among borderline personality and anxiety features in psychotherapy outpatients: An exploration of nomothetic and idiographic patterns.

Ellison¹,

Levy²,

Newman³

et al. 2020

Personality Disorders: Theory, Research, and Treatment

Self Cite

View full text Add to dashboard Cite

Borderline personality disorder (BPD) involves instability in self-concept, emotions, and behavior. However, the dynamic, longitudinal relations among BPD symptoms and between these symptoms and other problematic emotional experiences are poorly understood. It is also unclear whether these dynamics are the same across persons (including across diagnostic boundaries), specific to individuals with BPD, or idiographic. The current study uses ecological momentary assessment and Group Iterative Multiple Model Estimation (GIMME), a novel, datadriven approach to identifying dynamic patterns in time-series data at group, subgroup, and individual levels, to investigate the dynamic connections among select features of BPD (anger, impulsivity, and identity disturbance) and anxiety-related experiences. Forty-two psychiatric outpatients diagnosed with BPD (n = 27) or with an anxiety disorder, but not BPD (n = 15) rated their anger, identity disturbance, impulsivity, anxiety, stress, and calmness states six times per day for 21 days, providing a total of 4,699 surveys. Only one dynamic link between symptoms was identified that applied at the group level, and GIMME did not reveal stable subgroups of individuals with distinct symptom dynamics. Instead, these dynamics differed from individual to individual. These results suggest that connections among these BPD and anxiety symptoms do not depend on diagnosis and are somewhat idiographic. Case examples are used to illustrate the clinical utility of within-person symptom models as a supplement to traditional diagnostic information.

show abstract

The Questionable Ecological Validity of Ecological Momentary Assessment: Considerations for Design and Analysis

Cited by 95 publications

References 43 publications

Smartphone-Based Ecological Momentary Assessment of Well-Being: A Systematic Review and Recommendations for Future Studies

Smartphone-Based Ecological Momentary Assessment of Well-Being: A Systematic Review and Recommendations for Future Studies

Content validity and methodological considerations in ecological momentary assessment studies on physical activity and sedentary behaviour: a systematic review

Dynamics among borderline personality and anxiety features in psychotherapy outpatients: An exploration of nomothetic and idiographic patterns.

Contact Info

Product

Resources

About