Assessing Implementation Fidelity of a School-Based Crisis Prevention Program with an Ex-Post-Facto Design: The NETWASS FOI Assessment System

Abstract

This research paper presents an operationalization procedure for measuring fidelity of implementation (FOI) of a school-based crisis prevention program. The implementation literature recommends that program developers specify core components of an intervention that are directly related to a program's theory of change and need to be implemented with high FOI. This approach allows stakeholders some flexibility to adapt a program to individual circumstances yet helps assure that the intended outcomes are achieved. We trained 3,473 school staff in 98 German schools in the NETWorks Against School Shootings (NETWASS) program. Following the CORE cycle, we conceptualized 12 core components and operationalized relevant FOI dimensions of dosage, quality, adherence. and responsiveness. FOI was measured ex post facto, i.e after program implementation was completed, and separately for three distinct stages of implementation. Finally, we identified theoretical cutoffs for high fidelity on 15 measures using 91 items derived from an existing data set. Results indicate a high FOI across all schools for the dimensions of dosage and quality. Regarding responsiveness, high FOI was found for intervention compliance at t1 and program acceptance at the follow-up. Participant engagement during the trainings was measured separately and remained below our threshold. Adherence to 10 out of 12 core components was high. After training, school staff reported sufficient theoretical knowledge and were sensitized to recognize students in trouble, but actual case evaluation skills left room for improvement.The resulting FOI Assessment System requires validation by empirical research. Multi-level statistical modelling is necessary to test the hyothesized relationships between FOI per stage and outcomes, and obtain empirical validation of the hypothesized core components. Despite the methodological weaknesses of applying the CORE cycle ex post facto, it seems to be a feasible strategy to assess dimensions of FOI.