Evaluating investment in quality improvement capacity building: a systematic review

Abstract

PurposeLeading health systems have invested in substantial quality improvement (QI) capacity building, but little is known about the aggregate effect of these investments at the health system level. We conducted a systematic review to identify key steps and elements that should be considered for system-level evaluations of investment in QI capacity building.MethodsWe searched for evaluations of QI capacity building and evaluations of QI training programmes. We included the most relevant indexed databases in the field and a strategic search of the grey literature. The latter included direct electronic scanning of 85 relevant government and institutional websites internationally. Data were extracted regarding evaluation design and common assessment themes and components.Results48 articles met the inclusion criteria. 46 articles described initiative-level non-economic evaluations of QI capacity building/training, while 2 studies included economic evaluations of QI capacity building/training, also at the initiative level. No system-level QI capacity building/training evaluations were found. We identified 17 evaluation components that fit within 5 overarching dimensions (characteristics of QI training; characteristics of QI activity; individual capacity; organisational capacity and impact) that should be considered in evaluations of QI capacity building. 8 key steps in return-on-investment (ROI) assessments in QI capacity building were identified: (1) planning—stakeholder perspective; (2) planning—temporal perspective; (3) identifying costs; (4) identifying benefits; (5) identifying intangible benefits that will not be included in the ROI estimation; (6) discerning attribution; (7) ROI calculations; (8) sensitivity analysis.ConclusionsThe literature on QI capacity building evaluation is limited in the number and scope of studies. Our findings, summarised in a Framework to Guide Evaluations of QI Capacity Building, can be used to start closing this knowledge gap.

Assessing Quality Improvement Capacity in Primary Care Practices

“…opportunities to apply QI skills; exposure to further coaching/support in the workplace; a supportive organizational culture and leadership; and access to resources. 6 Cohen and colleagues highlighted the role of a complex interaction of internal and external factors on implementing improvement strategies in primary care. 53 Those factors include motivations of practice members to improve, the resources available, external influences on improvement options, and the perspective of practice members about where opportunities for improvement exist.…”

“…A recent systematic review of approaches to evaluate QI capacity found substantial variation in approaches to measure or observe QI capacity. 6 Specific to primary care, some tools and instruments exist that assess organizational characteristics more broadly associated with quality improvement [28][29][30][31][32] such as organizational culture, [33][34][35] or readiness for change. 36,37 However, the concept of QI capacity is distinct from practice culture or readiness.…”

Assessing Quality Improvement Capacity in Primary Care Practices

“…opportunities to apply QI skills; exposure to further coaching/support in the workplace; a supportive organizational culture and leadership; and access to resources. 6 Cohen and colleagues highlighted the role of a complex interaction of internal and external factors on implementing improvement strategies in primary care. 60 Those factors include motivations of practice members to improve, the resources available, external influences on improvement options, and the perspective of practice members about where opportunities for improvement exist.…”

“…4 QI capacity requires that teams of individuals who work in primary care settings possess an in-depth knowledge of approaches to QI, have the requisite skills and commitment to apply that knowledge to clinical care improvement, and are supported by clinical information systems in that effort. 5,6 A focus on building QI capacity/capability is one of the key characteristics of high performing health care organizations. [7][8][9] Smaller primary care practices comprise over half of all primary care practices in the U.S. 10 These smaller practices often struggle with improving their capacity to improve care quality.…”

“…A recent systematic review of approaches to evaluate QI capacity found substantial variation in approaches to measure or observe QI capacity and no organizational-level QI capacity building evaluations were found. 6 Specific to primary care, some tools and instruments exist that assess organizational characteristics more broadly associated with quality improvement [28][29][30][31][32] such as overall organizational maturity or development, 33 organizational culture, [34][35][36] or readiness for change. 37,38 However, the concept of QI capacity is distinct from organizational culture or readiness.…”