How Much Is Enough? Evaluating Intervention Implementation Efficiently

Abstract

This study investigated the use of brief observations to measure implementation of small group interventions using the Quality of Intervention Delivery and Receipt (QIDR) tool. Videos of 10-min segments representing the beginning, middle, and end of each 30-min intervention lesson were coded for implementation. Results indicated that (a) reliability can be achieved when observing a portion of a lesson; (b) QIDR scores obtained from 10-min segments are significantly correlated with the 30-min observations; and (c) post hoc analysis indicate the tool may be sensitive to changes in implementation quality at the intervention group level across time. Implications for research and practice are discussed, including the utility of shorter observations to increase the monitoring of intervention implementation to potentially improve outcomes.

Implementation Factors and Their Influence on Student Mathematics Outcomes

“…Educators should be evaluating implementation in practice to gain critical information regarding how to evaluate students' response to intervention. Such measures should be used to monitor instruction and provide feedback to practitioners to increase treatment adherence and implementation quality, as both have been found here and in previous work (Fritz et al, 2019) to relate to student outcomes. Specifically, higher treatment adherence and implementation quality were related to higher student outcomes.…”

“…Structural measures, including checklists, are more objective than process measures (Mowbray et al., 2003). Implementation quality measures can facilitate feedback to practitioners but tend to take longer to administer and are more process in nature (Fritz et al., 2019). Process measures, which are often evaluated through rating scales, can be more subjective in nature, and include interactions between the program staff and clients, treatment delivery, or program (Mowbray et al., 2003).…”

“…Since the implementation measures studied here are highly related to one another, practitioners could develop a more efficient observation schedule by interchanging these measures based on the observational purpose. For example, if feedback is necessary to improve practice or if the goal is to gain a qualitative understanding of intervention instruction, implementation quality measures may be necessary (Fritz et al., 2019; Harn, 2017). Conversely, if observation is occurring as a checkpoint before more in‐depth observations, then a treatment adherence measure may be more appropriate.…”

Current status and future directions in assessment of paraprofessional practices

“…Results indicated evidence of strong concurrent relationships between QIDR and CLASS (between 0.80 and 0.90), and a predictive relationship in which QIDR data accounted for approximately 36% of the group‐level variability in student reading outcomes. Reliability was examined in a subsequent analysis of the same data set by Fritz et al (2019): results included an intra‐class correlation of 0.71, indicating strong reliability in observations as short in duration as 10 min.…”