We develop an approach to evaluating frequentist model averaging procedures by considering them in a simple situation in which there are two‐nested linear regression models over which we average. We introduce a general class of model averaged confidence intervals, obtain exact expressions for the coverage and the scaled expected length of the intervals, and use these to compute these quantities for the model averaged profile likelihood (MPI) and model‐averaged tail area confidence intervals proposed by D. Fletcher and D. Turek. We show that the MPI confidence intervals can perform more poorly than the standard confidence interval used after model selection but ignoring the model selection process. The model‐averaged tail area confidence intervals perform better than the MPI and postmodel‐selection confidence intervals but, for the examples that we consider, offer little over simply using the standard confidence interval for θ under the full model, with the same nominal coverage.
Casella and Hwang, 1983, JASA, introduced a broad class of recentered confidence spheres for the mean θ of a multivariate normal distribution with covariance matrix σ 2 I, for σ 2 known. Both the center and radius functions of these confidence spheres are flexible functions of the data. For the particular case of confidence spheres centered on the positive-part James-Stein estimator and with radius determined by empirical Bayes considerations, they show numerically that these confidence spheres have the desired minimum coverage probability 1 − α and dominate the usual confidence sphere in terms of scaled volume. We shift the focus from the scaled volume to the scaled expected volume of the recentered confidence sphere. Since both the coverage probability and the scaled expected volume are functions of the Euclidean norm of θ, it is feasible to optimize the performance of the recentered confidence sphere by numerically computing both the center and radius functions so as to optimize some clearly specified criterion. We suppose that we have uncertain prior information that θ = 0. This motivates us to determine the center and radius functions of the confidence sphere by numerical minimization of the scaled expected volume of the confidence sphere at θ = 0, subject to the constraints that (a) the coverage probability never falls below 1 − α and (b) the radius never exceeds the radius of the standard 1 − α confidence sphere. Our results show that, by focusing on this clearly specified criterion, significant gains in performance (in terms of this criterion) can be achieved. We also present analogous results for the much more difficult case that σ 2 is unknown.
Volume 3 of Analysis of Messy Data by Milliken & Johnson (2002) provides detailed recommendations about sequential model development for the analysis of covariance. In his review of this volume, Koehler (2002) asks whether users should be concerned about the effect of this sequential model development on the coverage probabilities of confidence intervals for comparing treatments. We present a general methodology for the examination of these coverage probabilities in the context of the two-stage model selection procedure that uses two F tests and is proposed in Chapter 2 of Milliken & Johnson (2002). We apply this methodology to an illustrative example from this volume and show that these coverage probabilities are typically very far below nominal. Our conclusion is that users should be very concerned about the coverage probabilities of confidence intervals for comparing treatments constructed after this twostage model selection procedure.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.