The decision sampling framework: a methodological approach to investigate evidence use in policy and programmatic innovation

Abstract

Background Calls have been made for greater application of the decision sciences to investigate and improve use of research evidence in mental health policy and practice. This article proposes a novel method, “decision sampling,” to improve the study of decision-making and research evidence use in policy and programmatic innovation. An illustrative case study applies the decision sampling framework to investigate the decisions made by mid-level administrators when developing system-wide interventions to identify and treat the trauma of children entering foster care. Methods Decision sampling grounds qualitative inquiry in decision analysis to elicit information about the decision-making process. Our case study engaged mid-level managers in public sector agencies (n = 32) from 12 states, anchoring responses on a recent index decision regarding universal trauma screening for children entering foster care. Qualitative semi-structured interviews inquired on questions aligned with key components of decision analysis, systematically collecting information on the index decisions, choices considered, information synthesized, expertise accessed, and ultimately the values expressed when selecting among available alternatives. Results Findings resulted in identification of a case-specific decision set, gaps in available evidence across the decision set, and an understanding of the values that guided decision-making. Specifically, respondents described 14 inter-related decision points summarized in five domains for adoption of universal trauma screening protocols, including (1) reach of the screening protocol, (2) content of the screening tool, (3) threshold for referral, (4) resources for screening startup and sustainment, and (5) system capacity to respond to identified needs. Respondents engaged a continuum of information that ranged from anecdote to research evidence, synthesizing multiple types of knowledge with their expertise. Policy, clinical, and delivery system experts were consulted to help address gaps in available information, prioritize specific information, and assess “fit to context.” The role of values was revealed as participants evaluated potential trade-offs and selected among policy alternatives. Conclusions The decision sampling framework is a novel methodological approach to investigate the decision-making process and ultimately aims to inform the development of future dissemination and implementation strategies by identifying the evidence gaps and values expressed by the decision-makers, themselves.

Rapid-cycle systems modeling to support evidence-informed decision-making during system-wide implementation

“…Interviews documented a set of discrete and inter-related decisions required to promote implementation of trauma-informed screening. As reported elsewhere [ 41 ], implementation decisions with respect to trauma-informed screening were classified into five domains: Reach of the screening program , including which children to screen and at what ages. Content of the screening tool , including which screening tool to use, and whether it should directly assess traumatic life events, the sequelae of traumatic life events (e.g., symptoms), or both.…”

“…Given RCSM’s focus on the needs of decision-makers, an understanding of the organizational and interpersonal processes in place for decision-making is critical to determination of the appropriate sampling framework [ 41 ]. The first task is to identify the individuals who inform or make the decisions pertinent to the policy or programmatic domain of interest.…”

“…For example, our team relied on “decision sampling” to analyze the decisions confronted by mid-level policymakers. Based explicitly on decision analysis [ 53 ], the interview guide included questions on (1) decision points, (2) choices considered, (3) evidence and expertise regarding chance and outcomes, (4) outcomes prioritized, (5) expressed values, (6) tradeoffs considered in making the final decision, and (7) aspects of the decision-making process [ 41 ], itself. As detailed in a recent publication [ 41 ], decision sampling facilitated documentation of stakeholder questions and priorities, specifically through identification of specific questions relevant to actual decisions confronted by policymakers, which helped to articulate model purpose and scope.…”

“…As a remedy, theories of situated knowledge advocate for “critical interaction among the members of the scientific community [and] among members of different communities” [ 39 ] as the best way to discern scientific assumptions and address their potential consequences. Consistent with this focus, system dynamics is explicitly intended to help scientists uncover hidden assumptions and biases [ 40 ] based on recognition of the limits of traditional research methodologies as well as the observation that “we are not only failing to solve the persistent problems we face but are in fact causing them.” ( [ 28 ] , p.501) Recognizing the benefit of uncovering hidden assumptions and biases in our scientific understandings holds profound implications, shifting our translational efforts from uptake of research evidence alone to promoting the bidirectional exchange of evidence, expertise, and values [ 41 ].…”

Rapid-Cycle Systems Modeling to Support Evidence-Informed Decision-Making During System-Wide Implementation

“…Consistent with this focus, system dynamics is explicitly intended to help scientists uncover hidden assumptions and biases (40) based on recognition of the limits of traditional research methodologies as well as the observation that "we are not only failing to solve the persistent problems we face, but are in fact causing them." (28, p. 501) Recognizing the bene t of uncovering hidden assumptions and biases in our scienti c understandings holds profound implications, shifting our translational efforts from uptake of research evidence alone to promoting the bidirectional exchange of evidence, expertise, and values (41).…”