Psychometric assessment of three newly developed implementation outcome measures

Abstract

BackgroundImplementation outcome measures are essential for monitoring and evaluating the success of implementation efforts. Yet, currently available measures lack conceptual clarity and have largely unknown reliability and validity. This study developed and psychometrically assessed three new measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM).MethodsThirty-six implementation scientists and 27 mental health professionals assigned 31 items to the constructs and rated their confidence in their assignments. The Wilcoxon one-sample signed rank test was used to assess substantive and discriminant content validity. Exploratory and confirmatory factor analysis (EFA and CFA) and Cronbach alphas were used to assess the validity of the conceptual model. Three hundred twenty-six mental health counselors read one of six randomly assigned vignettes depicting a therapist contemplating adopting an evidence-based practice (EBP). Participants used 15 items to rate the therapist’s perceptions of the acceptability, appropriateness, and feasibility of adopting the EBP. CFA and Cronbach alphas were used to refine the scales, assess structural validity, and assess reliability. Analysis of variance (ANOVA) was used to assess known-groups validity. Finally, half of the counselors were randomly assigned to receive the same vignette and the other half the opposite vignette; and all were asked to re-rate acceptability, appropriateness, and feasibility. Pearson correlation coefficients were used to assess test-retest reliability and linear regression to assess sensitivity to change.ResultsAll but five items exhibited substantive and discriminant content validity. A trimmed CFA with five items per construct exhibited acceptable model fit (CFI = 0.98, RMSEA = 0.08) and high factor loadings (0.79 to 0.94). The alphas for 5-item scales were between 0.87 and 0.89. Scale refinement based on measure-specific CFAs and Cronbach alphas using vignette data produced 4-item scales (α’s from 0.85 to 0.91). A three-factor CFA exhibited acceptable fit (CFI = 0.96, RMSEA = 0.08) and high factor loadings (0.75 to 0.89), indicating structural validity. ANOVA showed significant main effects, indicating known-groups validity. Test-retest reliability coefficients ranged from 0.73 to 0.88. Regression analysis indicated each measure was sensitive to change in both directions.ConclusionsThe AIM, IAM, and FIM demonstrate promising psychometric properties. Predictive validity assessment is planned.Electronic supplementary materialThe online version of this article (doi:10.1186/s13012-017-0635-3) contains supplementary material, which is available to authorized users.

Implementation of eHealth Interventions Across the HIV Care Cascade: a Review of Recent Research

“…Standardized measures are increasingly available [49]. For example, three new 4-item measures of acceptability, feasibility, and appropriateness have recently demonstrated promising psychometric properties [50], and a 40-item measure of program sustainability has been adapted for use in LMIC settings [51, 52]. …”

A type II implementation-effectiveness hybrid quasi-experimental pilot study of a clinical intervention to re-engage people living with HIV into care, ‘Lost & Found’: an implementation science protocol

“…Nurses will complete a 38-item self-administered questionnaire (Additional file 4) to assess implementation outcomes i to iii at various time points (see Table 2), each of which is captured by a 5-item scale. Scales for feasibility and acceptability were inspired by Weiner et al's pragmatic measures and the TAPP-C (The Arson Prevention Program for Children) compatibility and complexity subscales [29,30]. These outcomes will be measured separately for each of the core intervention elements (i. identifying and documenting OOC patients, and ii.…”

Mapping the reach of a rural Transitions Nurse Program for veterans with geographic information systems

“…The design and reporting of the qualitative data from this evaluation were performed per the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist [11]. In addition to the qualitative data, the Acceptability of Intervention Measures and Appropriateness of Intervention Measures [12] survey was administered and analyzed using descriptive statistics.…”