Assessing Change in Intervention Research: The Benefits of Composite Outcomes

Abstract

Intervention research is often time- and resource-intensive, with numerous participants involved over extended periods of time. To maximize the value of intervention studies, multiple outcome measures are often included, either to ensure a diverse set of outcomes is being assessed or to refine assessments of specific outcomes. Here, we advocate for combining assessments, rather than relying on individual measures assessed separately, to better evaluate the effectiveness of interventions. Specifically, we argue that by pooling information from individual measures into a single outcome, composite scores can provide finer estimates of the underlying theoretical construct of interest while retaining important properties more sophisticated methods often forgo, such as transparency and interpretability. We describe different methods to compute, evaluate, and use composites depending on the goals, design, and data. To promote usability, we also provide a preregistration template that includes examples in the context of psychological interventions with supporting R code. Finally, we make a number of recommendations to help ensure that intervention studies are designed in a way that maximizes discoveries. A Shiny app and detailed R code accompany this article and are available at https://osf.io/u96em/ .

Causes and Consequences of Interindividual Response Variability: A Call to Apply a More Rigorous Research Design in Acute Exercise-Cognition Studies

“…In this regard, the size of the test battery (conducting several cognitive tests) could be of interest as it allows to calculate latent variables [an unobserved (latent) variable calculated based on the performance in a specific set of cognitive tests] and/or a composite score which can be of importance to draw further conclusions. The advantages and disadvantages of such an approach and further methodological aspects regarding cognitive testing (e.g., near and far transfer effect) are discussed elsewhere in more detail (Green et al, 2014(Green et al, , 2019Liu-Ambrose et al, 2018;Moreau and Wiebels, 2021). In this context, there is some evidence in the literature that cognitive tests showing reliable effects at the group level are not well-situated to detect individual differences (Draheim et al, 2021).…”

How Malleable Are Cognitive Abilities? A Critical Perspective on Popular Brief Interventions

“…12 addition, reciprocal relationships between cognitive ability and other factors, such as determination or self-regulation (Malanchini et al, 2019), create complex systems that are challenging to model, especially under traditional longitudinal frameworks (Watts, 2017) This inherent complexity suggests great potential for future work, away from effect-centered research and toward the development of theoretical frameworks of cognitive malleability (Katz et al, 2018). Formalizing hypotheses computationally might prove to be a necessary step to propose refined models that go beyond broad, uninformative claims of improvement, and enable individual-level predictions (Farrell & Lewandowsky, 2010;Moreau, 2021;Moreau & Wiebels, 2021). Similarly, refined measurements and explicit estimation of process overlap across the content of interventions and outcome measures will facilitate precise and robust inferences (Moreau & Wiebels, 2021).…”

“…Formalizing hypotheses computationally might prove to be a necessary step to propose refined models that go beyond broad, uninformative claims of improvement, and enable individual-level predictions (Farrell & Lewandowsky, 2010;Moreau, 2021;Moreau & Wiebels, 2021). Similarly, refined measurements and explicit estimation of process overlap across the content of interventions and outcome measures will facilitate precise and robust inferences (Moreau & Wiebels, 2021). These are timely issues for the field to consider-efforts toward building computational models of cognitive improvements are in line with recent calls to recognize the importance of theoretical work (Fiedler, 2018;Gray, 2017;Greenwald et al, 1986;Krakauer et al, 2017;Muthukrishna & Henrich, 2019;Smaldino, 2019;Szollosi et al, 2019) and the far-reaching ramifications of measurement issues (Eisenberg et al, 2019;Flake & Fried, 2019;Poldrack & Yarkoni, 2016) in psychology and neuroscience.…”

“…Finally, a major issue with subgroup analyses comes with the associated flexibility in data analysis; subgroups can be created in many different ways within a variable, and for many potential variables. In all cases, the solutions are straightforward-subgroup analyses should be predefined, ideally in the form of a preregistration; exploratory subgroup analyses should be clearly labeled as such, and confirmed in subsequent studies Moreau & Wiebels, 2021;Nosek et al, 2018). When many subgroup analyses are performed, they should preferably be combined in multilevel analyses, rather than assessed independently.…”

“…INSERT BOX 2 ABOUT HEREFinally, limited theoretical frameworks of cognitive malleability also preclude convincing claims for gains at the latent level, as opposed to the more plausible artificial improvements on a task or set of tasks(Moreau & Conway, 2014;Moreau & Wiebels, 2021; …”