Advancing the pragmatic measurement of sustainment: a narrative review of measures

Abstract

Background Sustainment, an outcome indicating an intervention continues to be implemented over time, has been comparatively less studied than other phases of the implementation process. This may be because of methodological difficulties, funding cycles, and minimal attention to theories and measurement of sustainment. This review synthesizes the literature on sustainment measures, evaluates the qualities of each measure, and highlights the strengths and gaps in existing sustainment measures. Results of the review will inform recommendations for the development of a pragmatic, valid, and reliable measure of sustainment. Methods A narrative review of published sustainment outcome and sustainability measures (i.e., factors that influence sustainment) was conducted, including appraising measures in the Society of Implementation Research Collaboration (SIRC) instrument review project (IRP) and the Dissemination and Implementation Grid-Enabled Measures database initiative (GEM-D&I). The narrative review used a snowballing strategy by searching the reference sections of literature reviews and definitions of sustainability and sustainment. Measures used frequently and judged to be comprehensive and/or validated by a team of implementation scientists were extracted for analysis. Results Eleven measures were evaluated. Three of the included measures were found in the SIRC-IRP, three in the GEM-D&I database, (one measure was in both databases) and six were identified in our additional searches. Thirteen constructs relating to sustainment were coded from selected measures. Measures covered a range of determinants for sustainment (i.e., construct of sustainability) as well as constructs of sustainment as an outcome. Strengths of the measures included, development by expert panels knowledgeable about particular interventions, fields or contexts, and utility in specific scenarios. A number of limitations were found in the measures analyzed including inadequate assessment of psychometric characteristics, being overly intervention or context specific, being lengthy and/or complex, and focusing on outer context factors. Conclusion There is a lack of pragmatic and psychometrically sound measures of sustainment that can be completed by implementation stakeholders within inner context settings (e.g., frontline providers, supervisors).

Provider REport of Sustainment Scale (PRESS): development and validation of a brief measure of inner context sustainment

“…The influences and strategies to promote an EBP’s sustainment may be unique or distinct from influences and strategies that are effective in previous stages and thus require additional tools and research approaches [ 7 , 8 ]. Despite acknowledgement of the importance of sustainment, there are multiple reasons why this final phase of the implementation process is less well studied, one of which being a lack of valid, easily accessible, pragmatic measures of sustainment that are appropriate for direct service providers [ 9 – 11 ].…”

“…Although there are a number of published measures of sustainment and sustainability, existing measures tend to address sustainability rather than sustainment and tend to focus on community public health programs (e.g., smoking cessation campaigns) rather than specific EBPs implemented at an organizational level [ 8 ]. In addition, a recent review of published measures of sustainment and sustainability found existing measures tend to be time-intensive, require stakeholders across multiple levels or stakeholders with knowledge across multiple levels to complete, and tend to be limited to assessing a particular EBP or only appropriate for a specific setting [ 9 ]. As an example, recently, a 35-item Sustainment Measurement System Scale (SMSS) [ 11 ] was published to assist Substance Abuse and Mental Health Services Administration (SAMHSA)-funded preventative programs.…”

“…Furthermore, they are often too lengthy and involved for many purposes, such as providing a quick and pragmatic assessment of the current state of sustainment. There appears to be no measure that is applicable across health care settings, that can be tailored for particular EBPs, and that is designed to be completed by frontline providers [ 9 ].…”

“…Our knowledge and understanding of sustainment would benefit from additional measures that are valid, reliable, suitable, and accessible indicators of whether an EBP continues to be delivered. Brief and pragmatic measures of the implementation process are needed to advance implementation theory, identify the mechanisms associated with the sustained use of EBPs, and measure implementation effectiveness [ 9 , 14 ]. This study is consistent with conceptual models and implementation frameworks such as the Exploration, Preparation, Implementation, Sustainment (EPIS) framework that identify multiple phases or stages in the implementation process [ 15 – 18 ] and include focus explicitly on the sustainment phase [ 19 , 20 ].…”

Recidivism Treatment for Justice-Involved Veterans: Evaluating Adoption and Sustainment of Moral Reconation Therapy in the US Veterans Health Administration

“…Specifically, adoption (corresponding to the Implementation phase of the EPIS framework) was defined as having started an MRT group after the initial training, and the adoption phase as the time period in which there was active support for the implementation of MRT (see “Training Program Overview” section). Sustainment was defined as maintenance of MRT after the active support ended (Moullin et al 2020 ). Accordingly, we (1) estimated the prevalence of MRT adoption and sustainment across VHA facilities, (2) identified facility characteristics associated with MRT adoption and sustainment, and (3) used qualitative approaches to understand the facilitators and barriers to sustainment of MRT across facilities.…”