Results from a psychometric assessment of a new tool for measuring evidence-based decision making in public health organizations

Abstract

Background In order to better understand how to improve evidence-based decision making (EBDM) in state health departments, measurement tools are needed to evaluate changes in EBDM. The purpose of this study was to test the psychometric properties of a new measurement tool to assess EBDM in public health practice settings. Methods A questionnaire was developed, pilot-tested and refined in an iterative process with the input of public health practitioners with the aim of identifying a set of specific measures representing different components of EBDM. Data were collected in a national survey of state health department chronic disease practitioners. The final dataset (n=879) for psychometric testing was comprised of 19 EBDM items that were first examined using exploratory factor analysis, and then confirmatory factor analysis. Results The final model from confirmatory factor analysis includes five latent factors representing components of EBDM: capacity for evaluation, expectations and incentives for EBDM, access to evidence and resources for EBDM, participatory decision making, and leadership support and commitment. Conclusions This study addresses the need for empirically tested and theory-aligned measures that may be used to assess the extent to which EBDM is currently implemented, and further, to gauge the success of strategies to improve EBDM, in public health settings. This EBDM measurement tool may help identify needed supports for enhanced capacity and implementation of effective strategies.

Data-driven quality improvement in low-and middle-income country health systems: lessons from seven years of implementation experience across Mozambique, Rwanda, and Zambia

“…Barriers to promoting evidence-informed decision making from high-income settings have included a lack of incentives, lack of funding, perception that policymakers are not interested in evidence-based practice, and perceived lack of expertise [ 38 – 40 ]. Furthermore, another recent study identified five latent factors of evidence-based decision making, including: capacity for evaluation, expectations and incentives, access to resources, participatory decision-making, and leadership support [ 41 ]. A number of these components mirror our observations through implementation experiences across Mozambique, Rwanda, and Zambia; however, future efforts should be undertaken to determine the extent to which psychometric properties of surveys and identified latent factors of evidence-based decision making differ across settings.…”

Controlling Chronic Diseases Through Evidence-Based Decision Making: A Group-Randomized Trial

“…Measures in the 65-item online Qualtrics Version January 2014–November 2016 (Qualtrics) survey were informed by a literature review ( 13 ) and earlier research by the study team ( 16 , 19 ). Measures, described in detail elsewhere ( 16 , 20 , 21 ), were tested with cognitive response methods and test–retest reliability ( 16 ). Survey questions assessed individual-level skills (eg, adapting interventions, action planning, communicating to policy audiences) and organizational-level capacities (eg, access to evidence, program evaluation, perceived supervisory expectations) (Appendix Table 2).…”

“…We used items from a 7-point Likert scale (1 = “strongly disagree” and 7 = “strongly agree”) to conduct exploratory factor analysis with orthogonal rotation to create individual scores for 5 factors: 1) access to research evidence and resources (4 items), 2) evaluation capacity (3 items); 3) supervisory expectations (3 items), 4) participatory decision making (3 items), and 5) agency leadership support (3 items) as in a previous national survey with state health department public health practitioners ( 21 ). By definition, the factor scores had a mean of zero and were normally distributed.…”

Assessing Organizational Supports for Evidence-Based Decision Making in Local Public Health Departments in the United States: Development and Psychometric Properties of a New Measure

“…Previously developed measures have focused on individual skills related to EBDM28 and organizational supports for EBDM within state health departments (SHDs) 29. No measures for organizational supports have been rigorously validated for use in LHDs; 1 existing measure for use in LHDs has limited reliability evidence only 30.…”