Enhancing Validity in Observational Settings When Replication is Not Possible

Abstract

We argue that political sciexntists can provide additional evidence for the predictive validity of observational and quasi-experimental research designs by minimizing the expected prediction error or generalization error of their empirical models. For observational and quasi-experimental data not generated by a stochastic mechanism under the researcher’s control, the reproduction of statistical analyses is possible but replication of the data-generating procedures is not. Estimating the generalization error of a model for this type of data and then adjusting the model to minimize this estimate—regularization—provides evidence for the predictive validity of the study by decreasing the risk of overfitting. Estimating generalization error also allows for model comparisons that highlight underfitting: when a model generalizes poorly due to missing systematic features of the data-generating process. Thus, minimizing generalization error provides a principled method for modeling relationships between variables that are measured but whose relationships with the outcome(s) are left unspecified by a deductively valid theory. Overall, the minimization of generalization error is important because it quantifies the expected reliability of predictions in a way that is similar to external validity, consequently increasing the validity of the study’s conclusions.

External Validity and Meta‐Analysis

“…Perhaps the most practical characterization of projectivism is Fariss and Jones (2018), who argue that the extrapolation of a single study's result should be viewed in terms of its predictive scope. But projectivist perspectives on external validity are not naturally suited for meta‐analyses, which essentially treats constituent studies symmetrically.…”

Spread of tweets in climate discussions: A case study of the 2019 Nobel Peace Prize announcement

“…This decreases the risk of overfitting an excessively complex model to noises in the sampled data, and consequently overestimating the significance of effects and reporting findings that will not replicate in the population (Babyak, 2004;Hawkins, 2004;McNeish, 2015). Consequently, this automatic variable selection process improves the parsimony and generalisability of the model and the validity of its interpretations (Fariss & Jones, 2018;McNeish, 2015). Further, a group lasso model performs variable selection in a grouped manner, so that each group of variables is included or excluded as a whole.…”

Endogenous rural dynamics: an analysis of labour markets, human resource practices and firm performance

“…This focus provides the advantage of controlling unwarranted excessive variance, spurious correlation and related biases: it also implies, however, that empirical generalizations of our results are not possible. Whether our empirical results hold true over variations in persons, settings, treatment variables and measurement variables (Shadish 2010, p. 4), and thereby generalize beyond the sample of firms from the food sector as expected by the theory and its auxiliary assumptions (Fariss and Jones 2018), will have to be reappraised with samples of other firms, industries and regions.…”