The Difference Between Instability and Uncertainty: Comment on Young and Holsteen (2017)

Abstract

Young and Holsteen (YH) introduce a number of tools for evaluating model uncertainty. In so doing, they are careful to differentiate their method from existing forms of model averaging. The fundamental difference lies in the way in which the underlying estimates are weighted. Whereas standard approaches to model averaging assign higher weight to better fitting models, the YH method weights all models equally. As I show, this is a nontrivial distinction, in that the two sets of procedures tend to produce radically different results. Drawing on both simulation and real-world examples, I demonstrate that in failing to distinguish between numerical variation and statistical uncertainty, the procedure proposed by YH will tend to overstate the amount of uncertainty resulting from variation across models. In standard circumstances, the quality of estimates produced using this method will tend to be objectively worse than that of conventional alternatives.

Specification analysis for technology use and teenager well-being: statistical validity and a Bayesian proposal

“…Under-weighting the right set of controls may lead to substantial biases. 7 To address this issue, BSCA displays a score (middle panel) for each control set (bottom panel), from most to least supported by the data, in addition to the corresponding coefficient estimates (top panel). The score is a transformation of the Extended Bayesian Information Criterion (EBIC) roughly interpretable as the probability of each control set given the data (Supplementary Information).…”

“…We briefly illustrate the above-mentioned issues (see Giudice & Gangestad and Selz for further discussion) 6,7 -and how BSCA addresses them -by re-assessing Orben & Przybylski's study. Our main contribution is providing a data analysis methodology that ameliorates pitfalls in SCA-based inference and hence amplifies its potential.…”

Null-hacking, a lurking problem

“…Young & Holsteen, 2017). Such simple averaging assumes equal weight among all models, in short, making the assumptions that all models are equal (see also Slez, 2017). Furthermore, such approaches not only treat all models as equal but all models as equally valid.…”

Comment: Some Challenges When Estimating the Impact of Model Uncertainty on Coefficient Instability

“…Since the model space is built on theory and past research, wouldn't we want to weight these models by their plausibility even without a diverse group of social scientists? If we weighted the plausibility of models, we would likely find that a plausibility-weighted model robustness procedure resulted in more "concentrated" results, to use the term found in Slez (2017). The plausibility-weighted computational model robustness analysis would likely result in more stable results than the unweighted procedure used by MY, even for the same set of plausible variables.…”