Individualization and Quality Improvement: Two New Scales to Complement Measurement of Program Fidelity

Abstract

Fidelity scales have been widely used to assess program adherence to the principles of an evidence-based practice, but they do not measure important aspects of quality of care. Pragmatic scales measuring clinical quality of services are needed to complement fidelity scales measuring structural aspects of program implementation. As part of the instrumentation developed for the National Implementing Evidence-Based Practices Project, we piloted a new instrument with two 5-item quality scales, Individualization (a client-level quality scale) and Quality Improvement (an organizational-level quality scale). Pairs of independent fidelity assessors conducted fidelity reviews in 49 sites in 8 states at baseline and at four subsequent 6-month intervals over a 2-year follow-up period. The assessors followed a standardized protocol to administer these quality scales during daylong site visits; during these same visits they assessed programs on fidelity to the evidence-based practice that the site was seeking to implement. Assessors achieved acceptable interrater reliability for both Individualization and Quality Improvement. Principal components factor analysis confirmed the 2-scale structure. The two scales were modestly correlated with each other and with the evidence-based practice fidelity scales. Over the first year, Individualization and Quality Improvement improved, but showed little or no improvement during the last year of follow-up. The two newly developed scales showed adequate psychometric properties in this preliminary study, but further research is needed to assess their validity and utility in routine clinical practice.

How to implement Illness Management and Recovery (IMR) in mental health service settings: evaluation of the implementation strategy

“…The General Organizational Index (GOI) is a 12-item scale measuring the general quality of the clinical care [32]. It consists of two subscales measuring quality improvement at the organizational level (i.e., existence of training and supervision facilities, process and outcome monitoring, and quality assurance) and at the consumer level (i.e., provision of individualized eligibility determination, assessment, treatment plan, treatment, and choice regarding service provision).…”

“…In addition, penetration (the extent to which the practice is offered) and understanding of and commitment to programme philosophy are measured. The scale has shown adequate psychometric properties [32]. …”

Barriers and facilitators in implementing a pilot, pragmatic, telemedicine-delivered healthy lifestyle program for obesity management in a rural, academic obesity clinic

“…To measure the inner setting, staff (n = 8) completed a 38-measure questionnaire [15] that evaluated culture, culture stress, culture effort, implementation climate, learning climate, leadership engagement, and available resources ( Table 2, Supplemental File #4). We assessed organizational change using an adapted version (Supplemental File #5) of the General Organizational Index [16] administered to staff. This scale is an interview of 11 domains with a 5-point rating of program philosophy, commitment, client eligibility and identification, health promotion plan and its treatment, training, process and outcome monitoring, assessment, quality assurance, choice supervision, and penetration.…”

Understanding the Components of Quality Improvement Collaboratives: A Systematic Literature Review

“…Policymakers are beginning to recognize that the minimalist training and consultation models typically used for large‐scale rollouts of new quality improvement practices are insufficient and that without monitoring, feedback, and organizational support, new practices often do not succeed (Bond et al ; Schoenwald and Hoagwood ; Schoenwald, Sheidow, and Letourneau ). The goal of our article is to update the literature on the QICs’ patient‐ and provider‐level outcomes, to characterize how the common QIC components are reported in empirical outcome studies, and to identify, to the extent possible, the relationships between the QICs’ components and provider‐ and patient‐level outcomes.…”