Rapid-Cycle Systems Modeling to Support Evidence-Informed Decision-Making During System-Wide Implementation

Abstract

Background: To “model and simulate change” is an accepted strategy to support implementation at scale. Much like a power analysis can inform decisions about study design, simulation models offer an analytic strategy to synthesize evidence that informs decisions regarding implementation of evidence-based interventions. However, simulation modeling is under-utilized in implementation science. To realize the potential of simulation modeling as an implementation strategy, additional methods are required to assist stakeholders to use models to examine underlying assumptions, consider alternative strategies, and anticipate downstream consequences of implementation. To this end, we propose Rapid-cycle Systems Modeling (RCSM)—a form of group modeling designed to promote engagement with evidence to support implementation. To demonstrate its utility, we provide an illustrative case study with mid-level administrators developing system-wide interventions that aim to identify and treat trauma among children entering foster care.Methods: RCSM is an iterative method that includes three steps per cycle: (1) identify and prioritize stakeholder questions, (2) develop or refine a simulation model, and (3) engage in dialogue regarding model relevance, insights, and utility for implementation. For the case study, 31 key informants were engaged in step 1, a prior simulation model was adapted for step 2, and six member-checking group interviews (n=16) were conducted for step 3.Results: Step 1 engaged qualitative methods to identify and prioritize stakeholder questions, specifically identifying a set of inter-related decisions to promote implementing trauma-informed screening. In step 2, the research team created a presentation to communicate key findings from the simulation model that addressed decisions about programmatic reach, optimal screening thresholds to balance demand for treatment with supply, capacity to start-up and sustain screening, and availability of downstream capacity to provide treatment for those with indicated need. In step 3, member-checking group interviews with stakeholders documented the relevance of the model results to implementation decisions, insight regarding opportunities to improve system performance, and potential to inform conversations regarding anticipated implications of implementation choices. Conclusions: By embedding simulation modeling in a process of stakeholder engagement, RCSM offers guidance to realize the potential of modeling not only as an analytic strategy, but also as an implementation strategy.

Using decision analysis to support implementation planning in research and practice

“…Importantly, decision analysis does not prescribe which kinds of consequences should be examined or what is right for a given context. The choice of what consequences to assess, how to value those impacts, and how the valuation leads to a decision is always context-specific and should incorporate the preferences and values of all and often diverse stakeholders [17,33,61,71]. Additionally, while many of the individual components of decision analysis may be familiar to implementation scientists (e.g., engaging stakeholders, selecting candidate implementation strategies), we believe it is valuable to situate these component pieces within a broader decision analysis framework.…”

“…Other times, the problem may be clear, and more effort is needed to assess the potential effects under each alternative or understand where major uncertainties are. This drives home that while the process may be constructed around coming to a decision, learning and insight gained throughout the decision analysis process can often be just as valuable [17,71], and once a process is completed for a given decision problem, the investment made can support subsequent decision-making.…”

“…The value of applying decision analytic approaches to implementation planning has been suggested in other work, though these methods remain underutilized [12,37,38,60,71]. To advance the use of decision analysis in implementation planning and research, we propose key areas for future research and practice based on issues considered in this paper and encourage discussion on how these suggestions can be prioritized by the field.…”

Implications of adverse childhood experiences screening on behavioral health services: A scoping review and systems modeling analysis.

“…A simulation model was developed to represent the process by which screening leads to delivery of behavioral health care services. Originally designed to address the needs of state-level policymakers working to implement trauma-informed screening, this model simulates demand for treatment that results from screening as well as workforce capacity to provide that treatment (Sheldrick, Stadnick, et al, 2019; Sheldrick et al, 2020). Below, we describe the model’s structure, outcomes, and parameters.…”

Future Directions in Lay Health Worker Involvement in Children’s Mental Health Services in the U.S.

“…Furthermore, simulation modeling has been proposed as an implementation strategy, in that it can guide planning among stakeholders about their assumptions about how a new innovation will impact the supply and demand of an intervention (Sheldrick et al, 2021). This may be an especially fitting approach to identify appropriate roles for LHWs to increase access to EBPs in low-resource settings, in that models can be developed to identify whether LHWs are needed to address supply or demand issues that are driving disparities McCollum et al, 2016).…”