An updated protocol for a systematic review of implementation-related measures

Abstract

BackgroundImplementation science is the study of strategies used to integrate evidence-based practices into real-world settings (Eccles and Mittman, Implement Sci. 1(1):1, 2006). Central to the identification of replicable, feasible, and effective implementation strategies is the ability to assess the impact of contextual constructs and intervention characteristics that may influence implementation, but several measurement issues make this work quite difficult. For instance, it is unclear which constructs have no measures and which measures have any evidence of psychometric properties like reliability and validity. As part of a larger set of studies to advance implementation science measurement (Lewis et al., Implement Sci. 10:102, 2015), we will complete systematic reviews of measures that map onto the Consolidated Framework for Implementation Research (Damschroder et al., Implement Sci. 4:50, 2009) and the Implementation Outcomes Framework (Proctor et al., Adm Policy Ment Health. 38(2):65-76, 2011), the protocol for which is described in this manuscript.MethodsOur primary databases will be PubMed and Embase. Our search strings will be comprised of five levels: (1) the outcome or construct term; (2) terms for measure; (3) terms for evidence-based practice; (4) terms for implementation; and (5) terms for mental health. Two trained research specialists will independently review all titles and abstracts followed by full-text review for inclusion. The research specialists will then conduct measure-forward searches using the “cited by” function to identify all published empirical studies using each measure. The measure and associated publications will be compiled in a packet for data extraction. Data relevant to our Psychometric and Pragmatic Evidence Rating Scale (PAPERS) will be independently extracted and then rated using a worst score counts methodology reflecting “poor” to “excellent” evidence.DiscussionWe will build a centralized, accessible, searchable repository through which researchers, practitioners, and other stakeholders can identify psychometrically and pragmatically strong measures of implementation contexts, processes, and outcomes. By facilitating the employment of psychometrically and pragmatically strong measures identified through this systematic review, the repository would enhance the cumulativeness, reproducibility, and applicability of research findings in the rapidly growing field of implementation science.Electronic supplementary materialThe online version of this article (10.1186/s13643-018-0728-3) contains supplementary material, which is available to authorized users.

Psychometric and Pragmatic Properties of Social Risk Screening Tools: A Systematic Review

“…Finally, the team abstracted information from each included article relevant to 14 psychometric and pragmatic properties using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). 13 PAPERS consists of the following 9 psychometric properties: internal consistency, convergent validity, discriminant validity, known-groups validity, predictive validity, concurrent validity, structural validity, responsiveness, and norms. The scale also includes the following 5 criteria for pragmatic properties: cost, accessibility of language, assessor burden (training), assessor burden (interpretation), and length.…”

“…Ideally, they are developed using gold standard processes that establish their reliability (i.e., the degree to which a tool measures the intended construct) and validity (i.e., the consistency of scores of items intended to measure the same constructs). 11,12 In addition, screening tools with favorable pragmatic properties (e.g., brief, easy to administer and score, and free or low cost) may promote implementation and translation of research into practice, 13,14 but developers traditionally have not approached the tool development process with both psychometric and pragmatic considerations in mind. 15 However, despite the rapid proliferation and implementation of social risk screening tools, 16,17 neither the psychometric nor pragmatic measurement qualities of these tools is well understood.…”

Implementation science made too simple: a teaching tool

“…Powell et al [3] conceptualized 73 discrete implementation strategies available for consideration when developing an implementation intervention. Proctor et al [4] offer 17 potential outcome domains to consider for an implementation study, and the Society for Implementation Research Collaboration has compiled a repository [5] of over 400 implementation-related measures.…”

Action, actor, context, target, time (AACTT): a framework for specifying behaviour

“…AACTT is compatible with any theory, model or framework in which behaviour is the focus of inquiry and thus has the potential to be integrated alongside key theoretical approaches used in the field. Given on-going efforts to systematise the measurement of theoretical constructs [54], and description of intervention components [41][42][43], there is a need for ensuring that the object of change, i.e. behaviour, is equally as carefully described and understood.…”