A Contextual Factors Framework to Inform Implementation and Evaluation of Public Health Initiatives

Abstract

Evaluating initiatives implemented across multiple settings can elucidate how various contextual factors may influence both implementation and outcomes. Understanding context is especially critical when the same program has varying levels of success across settings. We present a framework for evaluating contextual factors affecting an initiative at multiple phases of its life cycle, including design, implementation, scale-up, spread, and sustainability. After providing a brief overview of related frameworks from the fields of improvement science and implementation science and of the methods by which we drew from this literature to develop the current framework, we present how this framework was customized and applied to three national public health initiatives. We close with implications for how evaluators can apply the framework to facilitate a deeper understanding of a program's implementation and success, collaborate with project stakeholders, and facilitate sustainability, spread, and scale-up of public health initiatives.

Developing a conceptual framework for implementation science to evaluate a nutrition intervention scaled-up in a real-world setting

“…More than one-third of implementation science frameworks (n 13, Table 2) considered real-time evaluation. et al (44) To develop a framework to assess context and implementation of complex interventions Public health X Rapport et al (45) To reveal how implementation science is presented and understood in health services research contexts and clarify the foundational concepts Health service X X X X Vanderkruik et al (46) Present a framework for evaluating contextual factors affecting an initiative at multiple phases of its life cycle Public health X X Shediac-Rizkallah et al (47) To know what factors influence sustainability, provide strategy to fostering program sustainability and provide a future direction Public health X Schell et al (48) To present a new conceptual framework for program sustainability in public health Public health X Iwelunmor (49) To conduct a systematic review of empirical literature to explore how health interventions implemented in Sub-Saharan Africa are sustained Public health X Verhagen et al (50) To bridge the gap between knowledge derived from research and generate evidence-based usable information and tools for practice Sport science X X X • Understand what exactly is to be scaled-up to achieve large-scale impact (12) • Intervention characteristics: intervention sources, evidence strength and quality, relative advantage, adaptability, trialability, complexity, design quality and packaging (24) • Assess the nature and strength of the evidence and its potential for implementation (25) • Determining the impact of intervention through assessing reach and efficacy of an intervention implemented in a real-world setting by individuals who are not part of the original research (26) • Pre-conditionsidentifying need, target population and suitable intervention and pre-implementationintervention packaging and community input (28) • The effective implementation starts with the identification of relevant practice issues (problems or best practices) and matching research findings or guidelines (29) • Explore what neededconducting a comprehensive formative evaluation to determine the nature of what service or resource is needed (33) • Initial consideration regarding the host setting, conduct need assessment and readiness assessment (34) • Gather evidence that a technology or modality works from basic science as well as controlled assessments of effectiveness (38) • Explorationassess fit to ensure a useable innovation (43) • Evidence synthesis and describe the problem as encountered in practice in terms of problem magnitude, severity, societal burden and problem context (50) Implementation fidelity…”

“…• Require a culture of inquiry, evaluation and learning (10) • Consider interdisciplinary research to assess coverage, equity, utilisation, demand, outcomes and impacts (11) • Monitoring and evaluation, learning, accountability (12) • Context or environment into which the research is to be placed, and the method or way in which the process is facilitated (25) • Evaluating the interventions to assess positive and negative outcomes as we as public health impact (26) • Evaluation must measure all the factors listed above that influence the degree of implementation fidelity, such as intervention complexity and the adequacy of facilitation strategies (27) • For effective implementation, requires continuous evaluation and adapting plan (29) • Process evaluation to support the implementation effort effectively (34) • Conducting programme evaluation to inform policy (38) • Consider evolution to assess intervention effectiveness and economic evaluation: cost-benefit/ effectiveness budget impact (39) • Evaluation frameworks should be in place as part of new research design, ready to be applied at the appropriate stage in the implementation process (45) • Evaluation of contextual factors (46) • Evaluate within the RE-AIM Framework (reach, effectiveness, adoption, implementation and maintenance) (50) Course correction during implementation…”

The Growth of the Evaluation Tree in the Policy Analysis Forest: Recent Developments in Evaluation

“…Nevertheless, practical applications of systems thinking to evaluation are starting to emerge. They generally take the form of frameworks for understanding context (Vanderkruik & McPherson, ); system mapping and system dynamic approaches for describing the components of systems and how these connect and interact to generate change (e.g., stock‐and‐flow and causal loop diagrams); and the use of social network analysis to understand how “actors” within these systems are related (Williams & Hummelbrunner, ).…”

A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation

“…Planning frameworks that were solely for guidance on the design and development of an intervention were also excluded (e.g. [43][44][45], but a number were retained where they included guidance related to evaluation [46][47][48][49][50].…”

“…Fifty-two were stated as providing guidance on overall programme evaluation, nine as specific to process evaluation and one as specific to outcome evaluation. Several of the frameworks provided guidance on evaluating specific programme elements such as empowerment [83], partnerships and participation [68,78,80,81,87,92], contextual factors [50], or legacy [76]. Four frameworks were described as 'planning frameworks' but incorporated guidance on evaluation [46- Other frameworks that included guidance to facilitate both evaluation and planning, but were not specifically described as 'planning frameworks', e.g.…”