Time to get personal? The impact of researchers’ choices on the selection of treatment targets using the experience sampling methodology

Abstract

One of the promises of the experience sampling methodology (ESM) is that it could be used to identify relevant targets for treatment, based on a statistical analysis of an individual’s emotions, cognitions and behaviors in everyday-life. A requisite for clinical implementation is that outcomes of person-centered analyses are not wholly contingent on the researcher performing them. To evaluate how much researchers vary in their analytical approach and to what degree outcomes vary based on analytical choices, we crowdsourced the analysis of one individual patient’s ESM data to 12 prominent research teams, asking them what symptom(s) they would advise the treating clinician to target in subsequent treatment. The dataset was from a 25-year-old male with a primary diagnosis of major depressive disorder and comorbid generalized anxiety disorder, who completed momentary assessments related to depression and anxiety psychopathology prior to psychotherapy. Variation was evident at different stages of the analysis, from preprocessing steps (e.g., variable selection, clustering, handling of missing data) to the type of statistics. Most teams did include a type of vector autoregressive model, which examines relations between variables (e.g., symptoms) over time. Although most teams were confident their selected targets would provide useful information to the clinician, not one advice was similar: both the number (0-16) and nature of selected targets varied widely. This study makes transparent that the selection of treatment targets based on personalized models using ESM data is currently highly conditional on subjective analytical choices and highlights key methodological issues that need to be addressed in moving toward clinical implementation. Research proposal, data and materials: osf.io/h3djy/

A decennial review of psychotraumatology: what did we learn and where are we going?

“…Dynamic network analyses utilizing intensive longitudinal data may help to answer these questions, but few dynamic network PTSD studies have been published so far (Greene, Gelkopf, Epskamp, & Fried, 2018;Greene, Gelkopf, Fried, Robinaugh, & Lapid Pickman, 2019;Hoffart, Langkaas, Øktedalen, & Johnson, 2019), and there is a need for many more in this vein. Another important question is how to derive idiographic PTSD models in order to provide personalized interventions, given that recent work indicates that there is little agreement both about how to derive idiographic models of psychopathology, as well as the translation of these models to clinical recommendations (Bastiaansen et al, 2019). An additional challenge facing researchers is the development of analytic tools that successfully identify and separate out the common or core features of PTSD that are shared by all those with the disorder, from the more idiographic elements that vary from person to person.…”

Experience Sampling and Daily Diary Studies: Basic Concepts, Designs, and Challenges.pdf

“…Researcher degrees of freedom come into play when there are many different ways to analyze the same data set (even for the same research question) and researchers can flexibly choose among them. As data from ESM studies are comparatively complex, the possibilities to analyze (and pre-process) data are nearly endless (Bastiaansen et al, 2019). To illustrate this, consider the data set that we would have obtained using the design described earlier, assuming we had collected information on the person, such as their Big Five personality traits, using the BFI-2 (Soto & John, 2017).…”

Disentangling objective characteristics of learning situations from subjective perceptions thereof, using an experience sampling method design

“…While the methods proposed in this article attempt to contribute to further developments of personalised learning and interventions based on momentary assessments, it is important to keep in mind that much more research is needed to get from assessments to valid and helpful interventions. As Bastiaansen et al (2019) have shown, different teams of researchers can draw very different conclusions about needed interventions from the exact same intensive longitudinal dataset and its intra-individual analyses. Teams of software developers, methodologists, and educators will need to work together to identify valid and effective ways to draw conclusions about individual students' emotional needs for support from data like ours, and about the best ways to deliver appropriate interventions in the appropriate moments.…”