External Validity

Abstract

External validity captures the extent to which inferences drawn from a given study's sample apply to a broader population or other target populations. Social scientists frequently invoke external validity as an ideal, but they rarely attempt to make rigorous, credible external validity inferences. In recent years, methodologically oriented scholars have advanced a flurry of work on various components of external validity, and this article reviews and systematizes many of those insights. We first clarify the core conceptual dimensions of external validity and introduce a simple formalization that demonstrates why external validity matters so critically. We then organize disparate arguments about how to address external validity by advancing three evaluative criteria: model utility, scope plausibility, and specification credibility. We conclude with a practical aspiration that scholars supplement existing reporting standards to include routine discussion of external validity. It is our hope that these evaluation and reporting standards help rebalance scientific inquiry, such that the current obsession with causal inference is complemented with an equal interest in generalized knowledge.

Operationalizing the Journal of Human Lactation’s Gendered Language Policy

“…The external validity of all studies is a major concern relating to how useful a particular study may be for others who want to use the findings. "External validity captures the extent to which inferences drawn from a given study's sample apply to a broader population or other target populations" (Findley et al, 2021). If vague, non-precise, definitions are used to define a study's sample, there is a threat to external validity.…”

Wargaming for International Relations research

“…A flight simulator programmed with accurate real-world parameters, for instance, has high ecological validity and is a cheaper, easier, and safer way to train pilots and learn about their decision-making. 3 While literature across disciplines continues to debate its definition (Baumeister and Vohs, 2007; Brunswik, 1947: 276; Schmuckler, 2001), we consider ecological validity to be a key element of external validity—the generalizability of research findings beyond the research context (Findley et al, 2020). 4…”

Meta-Analyzing Individual Participant Data From Studies With Complex Survey Designs: A Tutorial on Using the Two-Stage Approach for Data From Educational Large-Scale Assessments

“…Using such measures, ELSAs allow researchers to describe distributions and patterns of relationships in a certain target population or subpopulation. ELSA data are therefore obtained from large, representative probability samples (Rutkowski et al, 2014), thus implying that descriptive analyses of these data meet the gold standard for obtaining reliable knowledge about vital educational research topics that can be generalized to well-defined populations (Findley et al, 2021;Shadish et al, 2002). More specifically, the random sampling process effectively responds to threats of sample selection bias because it ensures the representativeness of the sample on observable and unobservable dimensions of a certain well-defined population (Findley et al, 2021;Shadish et al, 2002).…”

“…ELSA data are therefore obtained from large, representative probability samples (Rutkowski et al, 2014), thus implying that descriptive analyses of these data meet the gold standard for obtaining reliable knowledge about vital educational research topics that can be generalized to well-defined populations (Findley et al, 2021;Shadish et al, 2002). More specifically, the random sampling process effectively responds to threats of sample selection bias because it ensures the representativeness of the sample on observable and unobservable dimensions of a certain well-defined population (Findley et al, 2021;Shadish et al, 2002). Further, given their large sample sizes, ELSAs provide very precise estimates of effect sizes for these populations (Rutkowski et al, 2014) with "effect size" broadly defined as any quantitative estimate used to address a certain research question (Kelley & Preacher, 2012).…”