Psychometric properties of implementation measures for public health and community settings and mapping of constructs against the Consolidated Framework for Implementation Research: a systematic review

Abstract

BackgroundRecent reviews have synthesised the psychometric properties of measures developed to examine implementation science constructs in healthcare and mental health settings. However, no reviews have focussed primarily on the properties of measures developed to assess innovations in public health and community settings. This review identified quantitative measures developed in public health and community settings, examined their psychometric properties, and described how the domains of each measure align with the five domains and 37 constructs of the Consolidated Framework for Implementation Research (CFIR).MethodsMEDLINE, PsycINFO, EMBASE, and CINAHL were searched to identify publications describing the development of measures to assess implementation science constructs in public health and community settings. The psychometric properties of each measure were assessed against recommended criteria for validity (face/content, construct, criterion), reliability (internal consistency, test-retest), responsiveness, acceptability, feasibility, and revalidation and cross-cultural adaptation. Relevant domains were mapped against implementation constructs defined by the CFIR.ResultsFifty-one measures met the inclusion criteria. The majority of these were developed in schools, universities, or colleges and other workplaces or organisations. Overall, most measures did not adequately assess or report psychometric properties. Forty-six percent of measures using exploratory factor analysis reported >50 % of variance was explained by the final model; none of the measures assessed using confirmatory factor analysis reported root mean square error of approximation (<0.06) or comparative fit index (>0.95). Fifty percent of measures reported Cronbach’s alpha of <0.70 for at least one domain; 6 % adequately assessed test-retest reliability; 16 % of measures adequately assessed criterion validity (i.e. known-groups); 2 % adequately assessed convergent validity (r > 0.40). Twenty-five percent of measures reported revalidation or cross-cultural validation. The CFIR constructs most frequently assessed by the included measures were relative advantage, available resources, knowledge and beliefs, complexity, implementation climate, and other personal resources (assessed by more than ten measures). Five CFIR constructs were not addressed by any measure.ConclusionsThis review highlights gaps in the range of implementation constructs that are assessed by existing measures developed for use in public health and community settings. Moreover, measures with robust psychometric properties are lacking. Without rigorous tools, the factors associated with the successful implementation of innovations in these settings will remain unknownElectronic supplementary materialThe online version of this article (doi:10.1186/s13012-016-0512-5) contains supplementary material, which is available to authorized users.

Indicators and Measurement Tools for Health Systems Integration: A Knowledge Synthesis

“…Future research should focus on developing, testing, and validating measures for all domains of health system integration. Similar observations have been made in other evolving fields that lack measures with robust psychometrics [153154]. Poor reporting may contribute to the quality gap, prompting some researchers to demand clear guidelines for reporting of survey research [155].…”

Examining fidelity in the INFORM trial, a complex team-based behavioural intervention

“…Fidelity measures tend to be intervention-speci c and may lack rigorous psychometric testing. [16,36,37] A strength of our study is multiple data collection methods to assess delity, including the gold standard -observation. [9] We established content validity of items using theory and expert agreement.…”

Examining the external validity of the CRUZA study, a randomized trial to promote implementation of evidence-based cancer control programs by faith-based organizations

“…Few standardized or validated measures exist of latent organizational characteristics associated with adoption or implementation of innovations [26,27]. When feasible, we adapted items from existing organizational surveys [19].…”