Reliability of a tool for measuring theory of planned behaviour constructs for use in evaluating research use in policymaking

Abstract

BackgroundAlthough measures of knowledge translation and exchange (KTE) effectiveness based on the theory of planned behavior (TPB) have been used among patients and providers, no measure has been developed for use among health system policymakers and stakeholders. A tool that measures the intention to use research evidence in policymaking could assist researchers in evaluating the effectiveness of KTE strategies that aim to support evidence-informed health system decision-making. Therefore, we developed a 15-item tool to measure four TPB constructs (intention, attitude, subjective norm and perceived control) and assessed its face validity through key informant interviews.MethodsWe carried out a reliability study to assess the tool's internal consistency and test-retest reliability. Our study sample consisted of 62 policymakers and stakeholders that participated in deliberative dialogues. We assessed internal consistency using Cronbach's alpha and generalizability (G) coefficients, and we assessed test-retest reliability by calculating Pearson correlation coefficients (r) and G coefficients for each construct and the tool overall.ResultsThe internal consistency of items within each construct was good with alpha ranging from 0.68 to alpha = 0.89. G-coefficients were lower for a single administration (G = 0.34 to G = 0.73) than for the average of two administrations (G = 0.79 to G = 0.89). Test-retest reliability coefficients for the constructs ranged from r = 0.26 to r = 0.77 and from G = 0.31 to G = 0.62 for a single administration, and from G = 0.47 to G = 0.86 for the average of two administrations. Test-retest reliability of the tool using G theory was moderate (G = 0.5) when we generalized across a single observation, but became strong (G = 0.9) when we averaged across both administrations.ConclusionThis study provides preliminary evidence for the reliability of a tool that can be used to measure TPB constructs in relation to research use in policymaking. Our findings suggest that the tool should be administered on more than one occasion when the intervention promotes an initial 'spike' in enthusiasm for using research evidence (as it seemed to do in this case with deliberative dialogues). The findings from this study will be used to modify the tool and inform further psychometric testing following different KTE interventions.

Results from a psychometric assessment of a new tool for measuring evidence-based decision making in public health organizations

“…It is also important to note that this works aligns with scientific literature based outside of the US, primarily in Canada, Australia and the U.K., which applies a related set of terminology and theoretical frameworks to understand strategies to close the gap between research evidence and public health policymaking and practice, such as knowledge translation and exchange (KTE) and evidence-informed decision making (EIDM) (Boyko et al, 2011; Dobbins et al, 2001; Dobbins et al, 2009; Lewin et al, 2012; Makkar et al, 2015; Peirson et al, 2012; Pettman et al, 2013; Squires et al, 2011; Waters, 2009; Yousefi-Nourei & Dobbins, 2015). Peirson et al (2012) identified several critical factors for building EIDM capacity at an organizational level in Canadian public health units that were similar to measures in the current instrument, such as leadership capacity, access to and resources for using evidence (including workforce skills), having a receptive organizational culture and knowledge management strategy.…”

Deliberative Dialogues as a Strategy for System-Level Knowledge Translation and Exchange

“…The forum conducts formative and summative evaluations of all their dialogues. The questionnaire that the forum uses includes general questions about the dialogue, specific questions about design features, questions about the intentions of participants to use what they learned (Boyko et al 2011) and questions about participants' roles and backgrounds. The evaluation questions require both ratings and written responses.…”

Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers’ capacity to engage with and use research

“…Tools designed for different levels of measurement (organisational, policy, individual) are needed to match the range of strategies employed; from organisation-wide efforts aiming to develop a receptive climate for using research through to individually-targeted professional development. Measures developed to date have focussed on organisational-level capacity [2, 24, 25], with only one instrument identified for individual-level measurement, and this focused on intention to use research rather than capacity [28] (see Additional file 1 for our analysis of these measures). While there are measures designed for clinical contexts (as reviewed by Squires et al [29], and more recent examples [30]), the content of these is tailored for health professionals and the measures have not been validated with policymakers.…”