Lack of group-to-individual generalizability is a threat to human subjects research

Abstract

Only for ergodic processes will inferences based on group-level data generalize to individual experience or behavior. Because human social and psychological processes typically have an individually variable and time-varying nature, they are unlikely to be ergodic. In this paper, six studies with a repeated-measure design were used for symmetric comparisons of interindividual and intraindividual variation. Our results delineate the potential scope and impact of nonergodic data in human subjects research. Analyses across six samples (with 87-94 participants and an equal number of assessments per participant) showed some degree of agreement in central tendency estimates (mean) between groups and individuals across constructs and data collection paradigms. However, the variance around the expected value was two to four times larger within individuals than within groups. This suggests that literatures in social and medical sciences may overestimate the accuracy of aggregated statistical estimates. This observation could have serious consequences for how we understand the consistency between group and individual correlations, and the generalizability of conclusions between domains. Researchers should explicitly test for equivalence of processes at the individual and group level across the social and medical sciences.

An Examination of Idiographic Networks of Posttraumatic Stress Disorder Symptoms

“…Additionally, multilevel analyses have estimated networks from longitudinal—but, nevertheless, aggregated—data (Greene et al., 2018; Hoffart et al., 2019). Such aggregations assume a degree of homogeneity in network structure that is likely untenable (Fisher et al., 2018) given the lack of statistical equivalence between inter‐ and intraindividual variability (Hamaker et al., 2005; Molenaar, 2004). Thus, insights from nomothetic networks are unlikely to adequately reflect processes that occur at the individual level.…”

“…That is, the variance and covariance of PTSD symptoms in cross‐sectional group data are not equivalent to the variance and covariance of these same symptoms for a single individual over time. Further, although some past research has estimated multilevel networks from longitudinal data as a way to address the limitations of cross‐sectional PTSD networks (Greene et al., 2018; Hoffart et al., 2019), these studies still fall short of describing the person‐specific nature of these intraindividual processes due to violations of the underlying homogeneity assumptions of these models (Fisher, Medaglia, & Jeronimus, 2018; Molenaar, 2005; Piccirillo & Rodebaugh, 2019). Thus, it follows that insights about the nature of PTSD gleaned from nomothetic networks are unlikely to adequately inform researchers and clinicians about processes occurring at the individual level.…”

Dysfunctional posttraumatic cognitions, posttraumatic stress and depression in children and adolescents exposed to trauma: a network analysis

“…A strength of our study is the use of diverse international trauma samples with a variety in age, trauma type and cultural background. Nevertheless, it is important to keep in mind that between-subject results might not generalize to within-subject levels (see Fisher, Medaglia, & Jeronimus, 2018).…”

An examination of idiographic networks of posttraumatic stress disorder symptoms

“…That is, the variance and covariance of PTSD symptoms in cross-sectional group data are not equivalent to the variance and covariance of these same symptoms for a single individual over time. Further, although some past research has estimated multilevel networks from longitudinal data as a way to address the limitations of crosssectional PTSD networks (Greene et al, 2018;Hoffart et al, 2019), these studies still fall short of describing the personspecific nature of these intraindividual processes due to violations of the underlying homogeneity assumptions of these models (Fisher, Medaglia, & Jeronimus, 2018;Molenaar, 2005;Piccirillo & Rodebaugh, 2019). Thus, it follows that insights about the nature of PTSD gleaned from nomothetic networks are unlikely to adequately inform researchers and clinicians about processes occurring at the individual level.…”