Why do evaluations of integrated care not produce the results we expect?

Abstract

A number of evaluations of models of integrated care have not produced the expected result of reduced hospital admissions, and in some cases have even found people receiving integrated care services using hospitals more than matched controls. We tested three hypotheses for these surprising results with a group of 50 integrated care experts in a seminar: (1) problems with the model; (2) problems of implementation; and (3) problems of evaluation. Our group of experts did not rule out any of these hypotheses and came up with some advice as to manage these issues. For example, model designers should rigorously test the underlying logic; commissioners should seek out advice from experts and patients/professionals; and evaluators should choose outcomes wisely, use mixed methods approaches, and provide regular feedback loops to implementation sites. Evaluating integrated care is a skilled task that requires multiple approaches in terms of the design and implementation of the models. National research funders or other appropriate bodies might consider developing an advisory service to provide support to local systems planning evaluations.

Key stakeholder experiences of an integrated healthcare pilot in Australia: a thematic analysis

“…We gained a nuanced understanding of their experiences which was further enhanced by conducting two rounds of interviews over the course of the WSICP. With integrated care programs frequently failing at the implementation phase [25], this thematic analysis of integrated care experiences and perspectives, and our regular reporting of the findings to WSICP management, informed the ongoing roll out of the WSICP. Resulting program changes were explored in later interviews.…”

“…Despite integrated care programs becoming more common, many do not achieve their expected results. Programs frequently fail at the implementation stage, and many do not plan early evaluation that informs their implementation [25].…”

Evaluating Large-Scale Integrated Care Projects: The Development of a Protocol for a Mixed Methods Realist Evaluation Study in Belgium

“…Moreover, studies often adopt an understanding attitude aiming for a formative instead of a summative evaluation. Managing this tension between researchers and commissioners -a well-known problem referenced in the international literature [71] -is challenging. Therefore, open and transparent discussions about mutual expectations, priorities, vision and the roles of each party should occur when the project begins and regularly afterward.…”

“…Mixed methods bring together qualitative and quantitative approaches into a single study and rely upon the complementary strengths of each approach to address the study questions [69]. Quadruple Aim outcomes must be considered ' distal outcomes' [70] and are not expected to change significantly in the first years after the start of the implementation [71]. In the meantime, the process indicators can be collected and analysed to explore the short-term changes.…”

Key stakeholder experiences of an integrated healthcare pilot in Australia: A thematic analysis

“…We gained a nuanced understanding of their experiences which was further enhanced by conducting two rounds of interviews over the course of the WSICP. With integrated care programs frequently failing at the implementation phase (25), this thematic analysis of integrated care experiences and perspectives, and our regular reporting of the ndings to WSICP management, informed the ongoing roll out of the WSICP. Resulting program changes were explored in later interviews.…”