Introduction The RAISE Connection Program Implementation and Evaluation study (RAISE-IES) developed tools necessary to implement and disseminate an innovative team-based intervention designed to promote engagement and treatment participation, foster recovery, and minimize disability among individuals experiencing early psychosis. This paper presents the treatment model and reports on service utilization and outcomes. Using a within-group analysis, it was hypothesized that individuals’ symptoms and functioning would improve over time. Methods A total of 65 individuals in RAISE Connection Program treatment across two sites, (Baltimore, MD and Manhattan, NY) were enrolled and received services for up to two years. Primary outcomes included social and occupational functioning as well as symptoms. Trajectories for individuals’ outcomes over time were examined using linear and quadratic mixed-effects models with repeated measures. Results Measures of occupational and social functioning improved significantly over time; symptoms declined, and rates of remission improved. Visits were most frequent during the first 3 months, with a mean of 23.1± 11.5 unduplicated staff encounters per quarter, decreasing to 8.8±5.2 such encounters in the final quarter of year 2. Conclusions The overall project was successful in that the treatment program was delivered and tools useful to other clinical settings were produced. The strengths of this study lie in the demonstrated feasibility of delivering the coordinated specialty care model and the associated high rates of engagement among individuals who are typically difficult to engage in treatment. Notwithstanding the lack of a built-in comparison group, participant outcomes were promising, with improvements comparable to those seen with other successful interventions.

Mixed methods research—i.e., research that draws on both qualitative and qualitative methods in varying configurations—is well suited to address the increasing complexity of public health problems and their solutions. This review focuses specifically on innovations in mixed methods evaluations of intervention, program or policy (i.e., practice) effectiveness and implementation. The article begins with an overview of the structure, function and process of different mixed methods designs and then provides illustrations of their use in effectiveness studies, implementation studies, and combined effectiveness-implementation hybrid studies. The article then examines four specific innovations: procedures for transforming (or “quantitizing”) qualitative data, applying rapid assessment and analysis procedures in the context of mixed methods studies, development of measures to assess implementation outcomes, and strategies for conducting both random and purposive sampling particularly in implementation-focused evaluation research. The article concludes with an assessment of challenges to integrating qualitative and quantitative data in evaluation research.

A large knowledge gap exists regarding the measurement of sustainability of evidence-based prevention programs for mental and behavioral health. We interviewed 45 representatives of 10 grantees and 9 program officers within 4 Substance Abuse and Mental Health Services Administration prevention grant initiatives to identify experiences with implementation and sustainability barriers and facilitators; what “sustainability” means and what it will take to sustain their programs; and which Consolidated Framework for Implementation Research (CFIR) elements are important for sustainability. Lists of sustainability determinants and outcomes were then compiled from each data set and compared with one another. Analysis of themes from interviews and free lists revealed considerable overlap between sustainability determinants and outcomes. Four sustainability elements were identified by all three data sets (ongoing coalitions, collaborations, and networks and partnerships; infrastructure and capacity to support sustainability; community need for program; and ongoing evaluation of performance and outcomes), and 11 elements were identified by two of three data sets (availability of funding; consistency with organizational culture; evidence of positive outcomes; development of a plan for implementation and sustainment; presence of a champion; institutionalization and integration of program; institutional support and commitment; community buy-in and support; program continuity; supportive leadership; and opportunities for staff training). All but one of the CFIR domain elements (pressure from other states, tribes, or communities) were endorsed as important to sustainability by 50% or more of participants. It may be more important to implement a standardized process of eliciting determinants and outcomes of sustainability than to implement a single standardized instrument.

Objective This study describes clients’ reports of factors that facilitated or impeded engagement in services offered by the Recovery After an Initial Schizophrenia Episode (RAISE) Connection Program for youth and young adults experiencing early psychosis and was part of the larger RAISE Implementation and Engagement Study (RAISE-IES). Method Semi-structured interviews with 32 clients and thematic qualitative analyses were used to examine participants’ experiences of program services, staff practices, their own engagement behaviors, and related factors such as expectations, family, illness, and setting. Results Clients’ statements indicated that central engagement factors include: services and staff interactions that are highly individualized, respectful, warm, and flexible; clients’ life goals being the center of services; family member engagement; personal attributes and program location and setting factors. Conclusions These interviews help explain the Connection Program’s effectiveness regarding client engagement and deepen our understanding of treatment engagement for youth and young adults experiencing early psychosis. The individualized, flexible, recovery-focused and assertive model of services and client-staff interaction, incorporating shared decision making and focus on client life goals should be implemented and sustained in services for this population.

Mental health programs can address many components of fidelity with routinely available data. Information from client interviews can be used to corroborate these administrative data. In an application of this approach, data from these sources indicated that a team-based intervention for people experiencing early psychosis was implemented as intended, including program elements related to shared decision-making as well as a range of evidence-based clinical interventions.

Background: Enhancing the sustainability of evidence-based prevention programs for mental and behavioral health requires tools for measuring both sustainability determinants and sustainment outcomes. The aim of this study was to develop the Sustainment Measurement System Scale (SMSS) and to assess its reliability and construct validity for measuring both determinants and outcomes of efforts to sustain prevention programs and initiatives. Methods: A 42-item scale comprised of items identified from qualitative data collected from 45 representatives of 10 programs and 8 SAMHSA program officers was administered to 186 representatives of 145 programs funded by 7 SAMHSA prevention grant initiatives. Cronbach's alphas were used to determine inter-item reliability. Convergent validity was assessed by comparisons of a global measure of sustainment with current SAMHSA-funding status and continued operation in the same form. Discriminant validity was assessed by comparisons of sustainability determinants with whether or not the program had undergone adaptations. Results: Confirmatory factor analysis provided support for a 35-item model fit to the data. Cronbach's alpha was .84 for the sustainment outcome construct and ranged from .70 to .93 for the sustainability determinant constructs. All of the determinant constructs were significantly associated with sustainment outcome individual and global measures for the entire sample (p < 0.01 to 0.001) and for community-based programs and programs with a substance abuse focus (p < 0.05 to 0.001). Convergent validity was supported by significant associations between the global sustainment measure and current SAMHSA funding status and continued operation in the same form (p < 0.001). Four of the sustainability determinant constructs (responsive to community needs; coalitions, partnerships, and networks; organizational staff capability; and evaluation, feedback, and program outcomes) were also significantly associated with current SAMHSA funding status (p < 0.5 to 0.01). With the exception of organizational staff capability, all sustainability determinants were unrelated to program adaptation as predicted. Conclusions: The SMSS demonstrated good reliability and convergent and discriminant validity in assessing likelihood of sustainment of SAMHSA funded prevention programs and initiatives. The measure demonstrates potential in identifying predictors of program sustainment and as a tool for enhancing the likelihood of successful sustainment through ongoing evaluation and feedback.

BackgroundSustaining prevention efforts directed at substance use and mental health problems is one of the greatest, yet least understood, challenges in the field of implementation science. A large knowledge gap exists regarding the meaning of the term “sustainment” and what factors predict or even measure sustainability of effective prevention programs and support systems.Methods/designThe U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) supports a diverse portfolio of prevention and treatment grant programs that aim to improve population and individual level behavioral health. This study focuses on four SAMHSA prevention grant programs, two of which target substance abuse prevention at the state or single community level, one targets suicide prevention, and one targets prevention of aggressive/disruptive behavior in elementary schools. An examination of all four grant programs simultaneously provides an opportunity to determine what is meant by the term sustainment and identify and support both the unique requirements for improving sustainability for each program as well as for developing a generalizable framework comprised of core components of sustainment across diverse prevention approaches. Based on an analysis of qualitative and quantitative data of 10 grantees supported by these four programs, we will develop a flexible measurement system, with both general and specific components, that can bring precision to monitoring sustainment of infrastructure, activities, and outcomes for each prevention approach. We will then transform this system for use in evaluating and improving the likelihood of achieving prevention effort sustainment. To achieve these goals, we will (1) identify core components of sustainment of prevention programs and their support infrastructures; (2) design a measurement system for monitoring and providing feedback regarding sustainment within the four SAMHSA’s prevention-related grant programs; and (3) pilot test the predictability of this multilevel measurement system across these programs and the feasibility and acceptability of a measurement system to evaluate and improve the likelihood of sustainment.DiscussionThis project is intended to improve sustainment of the supporting prevention infrastructure, activities, and outcomes that are funded by federal, state, community, and foundation sources.

Background A large knowledge gap exists regarding the measurement of sustainability of evidence-based prevention programs for mental and behavioral health. This paper describes an instrument that is a component of the Sustainment Measurement System (SMS), designed to assess both determinants and outcomes of efforts to sustain prevention programs and initiatives. Methods A 42-item scale was administered to 186 representatives of 145 programs funded by 7 SAMHSA prevention grant initiatives. Scale items were identified from qualitative data collected from 45 representatives of 10 programs and 8 SAMHSA program officers and organized into 9 categories of sustainment determinants and sustainment outcomes. Factor structure identified from principal axis factor analysis and elimination of items with CFA coefficients less than .300 resulted in a 35-item scale with 8 determinant factors and 1 outcome factor. Results Confirmatory Factor Analysis provided support for a 35-item model fit to the data. The SMSS demonstrated good inter-item reliability and all determinant factors were significantly associated with sustainment outcome individual and global measures. Conclusions The SMSS is easy to use and demonstrates good reliability and convergent and discriminant validity in assessing likelihood of sustainment of SAMHSA funded prevention programs and initiatives upon termination of original funding. The measure demonstrates potential in identifying predictors of program sustainment and as a tool for enhancing the likelihood of successful sustainment through ongoing evaluation and feedback.