Real-time evaluation pros and cons: Lessons from the Gavi Full Country Evaluation in Zambia

Abstract

The Full Country Evaluations were Gavi-funded real-time evaluations of immunisation programmes in Bangladesh, Mozambique, Uganda and Zambia, from 2013 to 2016. The evaluations focused on providing evidence for improvement of immunisation delivery in these countries and spanned all phases of Gavi support. The process evaluation approach of the evaluations utilised mixed methods to track progress against defined theories-of-change and related milestones during the various stages of implementation of the Gavi support streams. This article highlights complexities of this type of real-time evaluation and shares lessons learnt on conducting such evaluation from the Zambian experience. Real-time process evaluation is a complex evaluation methodology that requires sensitivity to the context of the evaluation, catering for various information needs of stakeholders, and establishment of mutually beneficial relationships between programme implementers and evaluators. When used appropriately, it can be an effective means of informing programme decisions and aiding programme improvement for both donors and local implementers.

Lessons Learned From Implementing Prospective, Multicountry Mixed-Methods Evaluations for Gavi and the Global Fund

“…In some contexts, CEPs joining meetings solely as observers was not acceptable—they were expected to contribute if they wanted to keep being invited—thus, they shifted into participant observers. 25 The Zambia FCE team highlighted their approach to provide meeting notes as a way of adding value, 15 and across all CEPs, evaluator reflexivity was used to balance independence and embeddedness. 11 , 26 – 28 Over time, the CEPs and stakeholders established shared expectations for engagement.…”

“…(Example FCE root cause analyses have been shared elsewhere. 11 , 15 ) Similarly, Tableau dashboards were a useful tool to support interpretation of quantitative data among team members with a range of quantitative data skillsets; all team members had access to the dashboards and would look at the quantitative results to generate questions for qualitative follow-up. For example, the PCE visualized quantitative data from Global Fund grant revisions to understand budgetary shifts, and then generated key informant interview questions to understand why the shifts occurred and how they were affecting implementation activities.…”

“…To clearly signal the strength of evidence, we developed a rubric informed by GRADE and other evidence rating systems 35 that rated the evidence along a 4-point scale. 15 However, while the GRADE rubric considers study design and rates randomized trials highly, our scale was limited to the types of evidence used in the FCE/PCE, so randomized trials were not feasible or fit-for-purpose. Our rubric considered the extent of triangulation between data sources and the quality of the sources.…”

“…Our rubric considered the extent of triangulation between data sources and the quality of the sources. Table 4 shows the strength of evidence rating used in the PCE, and Simuyemba et al 15 shared the rubric used in the FCE. Each finding was published with a rating to communicate our confidence in the conclusion, accounting for data quality and triangulation.…”

“…14 Our approach has shifted over time to reflect learnings gained through implementing these evaluations since 2013. 11 This article adds to the existing evaluation literature, and it expands on a complementary article from the Zambia FCE team's perspective 15 by taking a broader crosscountry view of lessons learned from 2 prospective mixed-methods evaluations. We present lessons learned across the evaluation life cycle to inform the implementation of ongoing or future complex evaluations.…”