Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact

Abstract

The question of why to evaluate a programme is seldom discussed in the literature. The present paper argues that the answer to this question is essential for choosing an appropriate evaluation design. The discussion is centered on summative evaluations of large-scale programme effectiveness, drawing upon examples from the fields of health and nutrition but the findings may be applicable to other subject areas. The main objective of an evaluation is to influence decisions. How complex and precise the evaluation must be depends on who the decision maker is and on what types of decisions will be taken as a consequence of the findings. Different decision makers demand not only different types of information but also vary in their requirements of how informative and precise the findings must be. Both complex and simple evaluations, however, should be equally rigorous in relating the design to the decisions. Based on the types of decisions that may be taken, a framework is proposed for deciding upon appropriate evaluation designs. Its first axis concerns the indicators of interest, whether these refer to provision or utilization of services, coverage or impact measures. The second axis refers to the type of inference to be made, whether this is a statement of adequacy, plausibility or probability. In addition to the above framework, other factors affect the choice of an evaluation design, including the efficacy of the intervention, the field of knowledge, timing and costs. Regarding the latter, decision makers should be made aware that evaluation costs increase rapidly with complexity so that often a compromise must be reached. Examples are given of how to use the two classification axes, as well as these additional factors, for helping decision makers and evaluators translate the need for evaluation--the why--into the appropriate design--the how.

Avaliação dos atributos da atenção primária à saúde em Chapecó, Brasil

“…Em Montes Claros-MG, pesquisa utilizando o PCATool-versão infantil, aplicado aos cuidadores de crianças, demonstrou um escore de acesso de (5,5), inferior ao Escore Geral da APS (6,6) 18 . Kolling, em estudo de linha de base dos municípios do Telessaúde do Rio Grande do Sul, utilizando a versão profissionais do PCATool, encontrou alto grau de orientação para a APS (7,2) e baixo grau no atributo de Acesso de Primeiro Contato (5,8) 19 . Outro estudo com o PCATool, nas UBS do município de Porto Alegre, com a aplicação do PCATool versão profissionais, encontrou resultados similares 20 .…”

“…Habicth et al 8 propuseram, baseados em Donabedian, que a avaliação de programas e de serviços de saúde buscasse orientar os responsáveis pela tomada de decisão (gestores em saúde). Diante de estratégias ou programas que já dispõem de evidências de plausibilidade e de probabilidade, o eixo de avaliação mais adequado para auxiliar os gestores em saúde seria a adequabilidade.…”

Participatory nutrition education and adoption of new feeding practices are associated with improved adequacy of complementary diets among rural Malawian children: a pilot study

“…As it was nonrandomized, we can evaluate the plausibility, but not the probability that the study intervention resulted in an improved adequacy of the complementary diet (Habicht et al, 1999). Hence, we cannot establish whether some other confounding factors contributed to the difference in dietary intakes between the two groups.…”

Strategies for delivering insecticide-treated nets at scale for malaria control: a systematic review

“…Retained papers underwent full-text review (performed independently by BW and LSP) to determine whether they described studies that satisfied the following criteria: subjects resided in areas where Plasmodium falciparum and Plasmodium vivax are endemic; ITN delivery at scale was evaluated; ITN ownership among households, receipt by pregnant women and/or use among children under 5 was evaluated; and an individual or cluster-randomized controlled design, a nonrandomized design, a quasi-experimental design, a beforeand-after design, an interrupted time series design or a cross-sectional design without temporal or geographical controls was used. [11][12][13] Papers meeting these criteria were termed "index papers". In addition to documenting and characterizing the strategies for delivering ITNs at scale and summarizing ITN ownership among households and ITN use among children under 5, this review also aimed to summarize the reported cost or costeffectiveness of different strategies and to synthesize information on reported factors influencing delivery of ITNs at scale.…”