Using Survival Analysis to Understand Patterns of Sustainment within a System-Driven Implementation of Multiple Evidence-Based Practices for Children’s Mental Health Services

Abstract

Evidence-based practice (EBP) implementation requires substantial resources in workforce training; yet, failure to achieve long-term sustainment can result in poor return on investment. There is limited research on EBP sustainment in mental health services long after implementation. This study examined therapists’ continued vs. discontinued practice delivery based on administrative claims for reimbursement for six EBPs [Cognitive Behavioral Interventions for Trauma in Schools (CBITS), Child–Parent Psychotherapy, Managing and Adapting Practices (MAP), Seeking Safety (SS), Trauma-Focused Cognitive Behavior Therapy (TF-CBT), and Positive Parenting Program] adopted in a system-driven implementation effort in public mental health services for children. Our goal was to identify agency and therapist factors associated with a sustained EBP delivery. Survival analysis (i.e., Kaplan–Meier survival functions, log-rank tests, and Cox regressions) was used to analyze 19 fiscal quarters (i.e., approximately 57 months) of claims data from the Prevention and Early Intervention Transformation within the Los Angeles County Department of Mental Health. These data comprised 2,322,389 claims made by 6,873 therapists across 88 agencies. Survival time was represented by the time elapsed from therapists’ first to final claims for each practice and for any of the six EBPs. Results indicate that therapists continued to deliver at least one EBP for a mean survival time of 21.73 months (median = 18.70). When compared to a survival curve of the five other EBPs, CBITS, SS, and TP demonstrated a higher risk of delivery discontinuation, whereas MAP and TF-CBT demonstrated a lower risk of delivery discontinuation. A multivariate Cox regression model revealed that agency (centralization and service setting) and therapist (demographics, discipline, and case-mix characteristics) characteristics were significantly associated with risk of delivery discontinuation for any of the six EBPs. This study illustrates a novel application of survival analysis to administrative claims data in system-driven implementation of multiple EBPs. Findings reveal variability in the long-term continuation of therapist-level delivery of EBPs and highlight the importance of both agency and workforce characteristics in the sustained delivery of EBPs. Findings direct the field to potential targets of sustainment interventions (e.g., strategic assignment of therapists to EBP training and strategic selection of EBPs by agencies).

Provider REport of Sustainment Scale (PRESS): development and validation of a brief measure of inner context sustainment

“…Brief and pragmatic measures of the implementation process are needed to advance implementation theory, identify the mechanisms associated with the sustained use of EBPs, and measure implementation effectiveness [ 9 , 14 ]. This study is consistent with conceptual models and implementation frameworks such as the Exploration, Preparation, Implementation, Sustainment (EPIS) framework that identify multiple phases or stages in the implementation process [ 15 – 18 ] and include focus explicitly on the sustainment phase [ 19 , 20 ].…”

Predicting the Sustained Implementation of an Evidence-Based Parenting Program: A Structural Equation Modelling Approach

“…Future studies should include more objective measures of program use (e.g. case files, rebate records; for example, see Brookman-Frazee et al, 2018;Lau et al, 2021), additional independent assessment of organisational capacity, as well as fidelity (e.g. observation).…”

Rates of sustainment in the Universal Stages of Implementation Completion

“…For example, a study by Ford et al examined the sustainment of the process improvement model NIATx200, finding that between 26.7 and 40.8% of clinics sustained at least one of the three initiated improvements 2 years after starting up, while 11.6% of clinics sustained for two of three improvements and 5% may have sustained all three [ 10 ]. Other studies may evaluate multiple evidence-based programs (EBPs) but differ on the levels of analysis: Brookman-Frazee et al examined sustainment on a therapist level within the Los Angeles County Mental Health Context, finding that the average length of time between the first and final claims for reimbursement (for the delivery of any EBP) was 21.71 months (with a standard deviation of 16.32 months) [ 11 ]; Bond et al examined sustainment on the site level, finding that 79.6% of sites in the National Implementing Evidence-Based Practices Project were sustained for 4 years following start-up, while 47% sustained 8 years following start-up [ 12 ]. Other evaluations relied on qualitative data alone [ 13 ] and unreliable self-reported data [ 14 , 15 ] or were unable to examine fidelity altogether [ 16 , 17 ].…”