Type I errors and power of the parametric bootstrap goodness‐of‐fit test: Full and limited information

Abstract

In sparse tables for categorical data well-known goodness-of-t statistics are not chi-square distributed. A consequence is that model selection becomes a problem. It has been suggested that a way out of this problem is the use of the parametric bootstrap. In this paper, the parametric bootstrap goodness-of-t test is studied by means of an extensive simulation study; the Type I error rates and power of this test are studied under several conditions of sparseness. In the presence of sparseness, models were used that were likely to violate the regularity conditions. Besides bootstrapping the goodness-of-t usually used (full information statistics), corrected versions of these statistics and a limited information statistic are bootstrapped. These bootstrap tests were also compared to an asymptotic test using limited information. Results indicate that bootstrapping the usual statistics fails because these tests are too liberal, and that bootstrapping or asymptotically testing the limited information statistic works better with respect to Type I error and outperforms the other statistics by far in terms of statistical power. The properties of all tests are illustrated using categorical Markov models.

Synthetic Statistical Approach Reveals a High Degree of Richness of Microbial Eukaryotes in an Anoxic Water Column

“…First, we assess goodness of fit, using a chi-square statistic computed across all cells (frequency counts) as a simple measure of discrepancy, and also using a chi-square statistic computed after concatenating cells for a minimum expected cell count of 5 in order to obtain an asymptotically correct goodness-of-fit test. (Currently we are implementing a computationally intensive goodness-of-fit assessment that avoids the necessity of concatenating cells [37], and we are adding the AIC to our computations for comparison of several models at a fixed value of , which is sometimes of interest.) We augment this numeric analysis with careful point-by-point examination of model fit, especially to the rare frequency counts.…”

Limited Information Goodness-of-fit Testing in Multidimensional Contingency Tables

“…Also, pooling cells ad hoc after the model has been fitted may result in a test statistic with an unknown asymptotic null sampling distribution. Regarding (b), generating the empirical sampling distribution of the goodness-of-fit statistic using a resampling method such as the parametric bootstrap method (e.g., Collins, Fidler, Wugalter, & Long, 1993;Bartholomew & Tzamourani, 1999) may result in trustworthy p-values (but see Tollenaar & Mooijaart, 2003). However, resampling methods may be very time-consuming if the researcher is interested in comparing the fit of several models.…”

A Cautionary Note on Using G2(dif) to Assess Relative Model Fit in Categorical Data Analysis

“…Maybe for this reason, to date the use of resampling to test models for categorical data is not widespread. Furthermore, a recent simulation study by Tollenaar and Mooijart (2003) revealed that the p-values for x 2 and G 2 obtained using bootstrap need not be accurate.…”