Improving Design Sensitivity through Intervention-Sensitive Measures

Abstract

Despite evidence that the choice of dependent measures can significantly influence design sensitivity, many evaluators default to traditional measures that may be insensitive to intervention effects. This paper describes an innovative set of test development guidelines designed to select items and create aggregate scales that are better able to detect program effects. The application of these Intervention Item Selection Rules (IISRs) is illustrated during the initial development of an outcome measure, completed by teachers, for elementary age children receiving psychosocial services from community mental health agencies. The major scale formed with these change-sensitive items displayed a larger effect size and an adequate reliability estimate.

The Systemic Therapy Inventory of Change (STIC) Initial Scales: Are they sensitive to change?

“…That is, as many therapists would expect, the IPS but not the BDI-II, BAI, or OQ-45, reveals that psychotherapy clients appear to express their feelings more openly and gain a greater understanding of and insight into their behavior and themselves throughout therapy. One reason for differences in the sensitivity of measures to intervention effects is that some measures may be more or less to assess the multiple effects of an intervention (Meier, 2004). Therefore, the IPS may make a unique contribution to the measurement of individual functioning over time by tapping these dimensions that the BDI-II, BAI, or OQ-45 are not strongly saturated with and detecting the multiple effects of psychotherapy.…”

Is it possible to integrate psychotherapy theory on the basis of empirical findings? The disruptive role of method effects.

“…Can test items, subscales, and scales detect specific changes that are expected (e.g., decreases in conduct problems in children as a result of an anger management group) as well as general changes (e.g., decreases in negative affect, such as anxiety and depression, associated with a wide variety of diagnoses and presenting problems)? Meier (2000Meier ( , 2004 proposed a set of Intervention Item Selection Rules (IISRs; cf. Gresham et al, 2010;Vermeersch et al, 2004) designed to identify intervention-sensitive items.…”

“…Intervention-sensitive items should remain relatively stable over time when no intervention is present (i.e., be reliable), but should evidence change in response to an intervention (i.e., evidence state effects). Research with progress monitoring and outcome measures developed using change-sensitive criteria indicates that change-sensitive measures possess different psychometric characteristics compared with tests developed through traditional approaches (Meier, 2000(Meier, , 2004Meier, McDougal, & Bardos, 2008;Weinstock & Meier, 2003). In general, scores on treatment-sensitive measures show larger effect sizes, when completed by individuals receiving psychosocial interventions, than scores on tests developed through traditional procedures.…”

“…Meier (2000, 2004) proposed a set of Intervention Item Selection Rules (IISRs; cf. Gresham et al, 2010; Vermeersch et al, 2004) designed to identify intervention-sensitive items.…”

Preventive Mental Health at School

“…Tools created with this in mind can demonstrate change as a result of a behavioral intervention and can also examine relative change across groups. Meir ( 2004 ) points out that traditional measures, such as the commonly used commercial rating scales, may be insensitive to intervention effects. Meir proposed a process of "Intervention Item Selection Rules" (IISRs) when developing outcome measures, which would display larger effect sizes along with a reliability estimate that would be acceptable.…”