Robustness and power of analysis of covariance applied to ordinal scaled data as arising in randomized controlled trials

Abstract

In clinical trials comparing two treatments, ordinal scales of three, four or five points are often used to assess severity, both prior to and after treatment. Analysis of covariance is an attractive technique, however, the data clearly violate the normality assumption and in the presence of small samples, and large sample theory may not apply. The robustness and power of various versions of parametric analysis of covariance applied to small samples of ordinal scaled data are investigated through computer simulation. Subjects are randomized to one of two competing treatments and the pre-treatment, or baseline, assessment is used as the covariate. We compare two parametric analysis of covariance tests that vary according to the treatment of the homogeneity of regressions slopes and the two independent samples t-test on difference scores. Under the null hypothesis of no difference in adjusted treatment means, we estimated actual significance levels by comparing observed test statistics to appropriate critical values from the F- and t-distributions for nominal significance levels of 0.10, 0.05, 0.02 and 0.01. We estimated power by similar comparisons under various alternative hypotheses. The model which assumes homogeneous slopes and the t-test on difference scores were robust in the presence of three, four and five point ordinal scales. The hierarchical approach which first tests for homogeneity of regression slopes and then fits separate slopes if there is significant non-homogeneity produced significance levels that exceeded the nominal levels especially when the sample sizes were small. The model which assumes homogeneous regression slopes produced the highest power among competing tests for all of the configurations investigated. The t-test on difference scores also produced good power in the presence of small samples.

Untitled

“…Sullivan and D'Agostino [26] describe a test as 'robust' if the actual significance level does not exceed 10% of the nominal significance level (e.g. less than or equal to 0.055 when the nominal significance level is 0.05).…”

Surgical Outcomes for Velopharyngeal Insufficiency in Velocardiofacial Syndrome and Nonsyndromic Patients

“…The mean and 95% confidence interval of pre-and postoperative speech parameters (speech intelligibility, VPI severity, speech volume, nasal air emission, and resonance) were calculated for the non-VCFS and VCFS groups. Repeated measures analysis of variance (Sullivan and D'Agostino, 2003) was performed for each group on the above speech parameters at pre-and postoperative time points in order to measure statistical significance, set at p Ͻ .05. An analysis also was performed to detect which individuals achieved a 2-point improvement in speech parameters.…”

Parent–child agreement across child health-related quality of life instruments: a review of the literature

“…However, statistical studies of HRQL measures have suggested that many tests that rely on normality are in fact robust to violations of normality, but only if the sample size is large enough relative to the departure from normality [82]. However, how small is too small is unclear, with some authors suggesting the t-test to be robust with samples as small as 20 [83] and others cautioning against using any parametric statistics with samples of under 100 [82]. Thus low parent-child correlations may not always reflect a lack of agreement.…”