Comparing denominator degrees of freedom approximations for the generalized linear mixed model in analyzing binary outcome in small sample cluster-randomized trials

Abstract

BackgroundSmall number of clusters and large variation of cluster sizes commonly exist in cluster-randomized trials (CRTs) and are often the critical factors affecting the validity and efficiency of statistical analyses. F tests are commonly used in the generalized linear mixed model (GLMM) to test intervention effects in CRTs. The most challenging issue for the approximate Wald F test is the estimation of the denominator degrees of freedom (DDF). Some DDF approximation methods have been proposed, but their small sample performances in analysing binary outcomes in CRTs with few heterogeneous clusters are not well studied.MethodsThe small sample performances of five DDF approximations for the F test are compared and contrasted under CRT frameworks with simulations. Specifically, we illustrate how the intraclass correlation (ICC), sample size, and the variation of cluster sizes affect the type I error and statistical power when different DDF approximation methods in GLMM are used to test intervention effect in CRTs with binary outcomes. The results are also illustrated using a real CRT dataset.ResultsOur simulation results suggest that the Between-Within method maintains the nominal type I error rates even when the total number of clusters is as low as 10 and is robust to the variation of the cluster sizes. The Residual and Containment methods have inflated type I error rates when the cluster number is small (<30) and the inflation becomes more severe with increased variation in cluster sizes. In contrast, the Satterthwaite and Kenward-Roger methods can provide tests with very conservative type I error rates when the total cluster number is small (<30) and the conservativeness becomes more severe as variation in cluster sizes increases. Our simulations also suggest that the Between-Within method is statistically more powerful than the Satterthwaite or Kenward-Roger method in analysing CRTs with heterogeneous cluster sizes, especially when the cluster number is small.ConclusionWe conclude that the Between-Within denominator degrees of freedom approximation method for F tests should be recommended when the GLMM is used in analysing CRTs with binary outcomes and few heterogeneous clusters, due to its type I error properties and relatively higher power.

Review of Recent Methodological Developments in Group-Randomized Trials: Part 2—Analysis

“…32–34 For binary outcomes, alternative methods for specifying the test degrees of freedom have been examined in small sample GRTs and the between-within method is recommended. 32,35 …”

Comparing the resistance of mid‐maturing maize varieties to European corn borer (Ostrinia nubilalis Hbn.)—results from the Polish VCU registration field trials

“…For this reason, in this paper the variety effects were tested using Wald F test implemented in GenStat. Li and Redden () in their studies showed that Kenward–Roger method controls the type I error, but it is more conservative than the between‐within denominator degrees of freedom approximation method (see Li & Redden, for further details). To test the significance of the fixed effects, one can also use the parametric bootstrap (see Halekoh & Højsgaard, ; or Warton & Hui, ), but from our experience for large data sets, this procedure can be time‐consuming.…”

No Need to Turn Bayesian in Multilevel Analysis with Few Clusters: How Frequentist Methods Provide Unbiased Estimates and Accurate Inference

“…Using the t-distribution for inference about upper-level e ects is not straightforward due to the fact that the appropriate degrees of freedom are unknown in the context of mixed e ects models. Statisticians have proposed several approximations to the e ective degrees of freedom over the past decades (c.f., Schaalje, McBride, and Fellingham, 2002;Manor and Zucker, 2004;Li and Redden, 2015). Some of these are clearly inadequate for inference about direct context e ects and cross-level interactions, because they will typically result in degrees of freedom that far exceed the number of clusters.…”

“…We therefore do not include the so-called Kenward-Roger approximation (Kenward and Roger, 1997) in our simulations, a computationally more intensive alternative that is closely related to the Satterthwaite approximation and yields almost identical results in the single-parameter (or, more generally, single-constraint) case (Li and Redden, 2015). Practitioners interested in jointly testing multiple constraints (e.g., the hypothesis that all coe cients except for the intercept are zero) should consider the Kenward-Roger approximation and we hope that our contribution stimulates research on their relative performance in the kinds of "large n, small m" settings that are typical of comparative politics and adjacent disciplines.…”