Evaluating clinical significance through equivalence testing: Extending the normative comparisons approach

Abstract

The field of psychology, as with many other disciplines, has been increasingly interested in being able to measure the effectiveness of behavioral interventions. This trend has led to a number of different approaches for measuring clinical significance, each addressing a slightly different aspect of the clinical outcome. Recently, clinical psychologists (and clients) have supported the contention that one of the most important therapeutic questions is whether clients are functioning equivalently to normal controls following an intervention. To address this question, Kendall, Marrs-Garcia, Nath, and Sheldrick (1999) presented an approach to measuring clinical significance that utilizes tests of equivalence. The present study clarifies the nature of the hypotheses being conducted in measuring clinical significance with tests of equivalence and extends the approach by incorporating recent advances in equivalence testing. A revised approach for evaluating clinical significance via equivalence testing is proposed, and an empirical example demonstrating this approach is provided.

Psychological well-being in obese inpatients with ischemic heart disease at entry and at discharge from a four-week cardiac rehabilitation program

“…Equivalence tests were conducted using normative comparisons (Kendall et al, 1999; Cribbie and Arpin-Cribbie, 2009) to assess the clinical significance of statistically significant improvements. More specifically, normative comparisons evaluate whether an intervention that has produced a statistically significant improvement in the outcome has also produced a clinically meaningful effect, by determining if the treated group is equivalent to a normal comparison group after the intervention.…”

Group Level Clinical Significance: An Analysis of Current Practice

“…Of those that correctly used the method, all used the Schuirmann (1987) two one-sided testing approach for assessing the equivalence of the treated and normal comparison groups. One concern with the use of this method is that it is not robust to unequal sample sizes and variances (Cribbie and Arpin-Cribbie 2009;Gruman, Cribbie, and Arpin-Cribbie 2007). This is especially problematic given that the group sizes and standard deviations of normal comparison and treated clinical groups often differ substantially.…”

Assessing clinical significance using robust normative comparisons

“…Gruman et al (2007) found that Type I error rates for the Schuirmann-Welch test are maintained at approximately α even when sample sizes and variances are extremely unequal. There is also very little power lost by using the Schuirmann-Welch procedure instead of the original Schuirmann equivalence testing procedure when sample sizes and variances are equal (Cribbie & Arpin-Cribbie, 2009). Therefore, due to its robustness when the sample variances are heterogeneous, it was recommended that researchers evaluating clinical significance via equivalence testing routinely utilize the SchuirmannWelch procedure.…”

“…It is well known, however, that sample sizes and variances of the normative and clinical groups are regularly very disparate (Cribbie & Arpin-Cribbie, 2009). Furthermore, as Gruman, Cribbie, and Arpin-Cribbie (2007) show, empirical Type I error rates for Schuirmann's test of equivalence have been found to deviate substantially from the nominal α level when sample sizes and variances are unequal.…”

“…The Schuirmann-Yuen (van Wieringen & Cribbie, in press), Schuirmann-Welch, and Schuirmann (Cribbie & Arpin-Cribbie, 2009) equivalence tests for normative comparisons were modeled with the R statistical software application (R Foundation for Statistical Computing, 2011). For our normative comparison analyses, we defined δ in the equivalence interval (δ, δ) as one standard deviation of the normative mean.…”