BackgroundIn recent years, several large studies have assessed the costs of national infant immunization programs, and the results of these studies are used to support planning and budgeting in low- and middle-income countries. However, few studies have addressed the costs and cost-effectiveness of interventions to improve immunization coverage, despite this being a major focus of policy attention. Without this information, countries and international stakeholders have little objective evidence on the efficiency of competing interventions for improving coverage.MethodsWe conducted a systematic literature review on the costs and cost-effectiveness of interventions to improve immunization coverage in low- and middle-income countries, including both published and unpublished reports. We evaluated the quality of included studies and extracted data on costs and incremental coverage. Where possible, we calculated incremental cost-effectiveness ratios (ICERs) to describe the efficiency of each intervention in increasing coverage.ResultsA total of 14 out of 41 full text articles reviewed met criteria for inclusion in the final review. Interventions for increasing immunization coverage included demand generation, modified delivery approaches, cash transfer programs, health systems strengthening, and novel technology usage. We observed substantial heterogeneity in costing methods and incompleteness of cost and coverage reporting. Most studies reported increases in coverage following the interventions, with coverage increasing by an average of 23 percentage points post-intervention across studies. ICERs ranged from $0.66 to $161.95 per child vaccinated in 2017 USD. We did not conduct a meta-analysis given the small number of estimates and variety of interventions included.ConclusionsThere is little quantitative evidence on the costs and cost-effectiveness of interventions for improving immunization coverage, despite this being a major objective for national immunization programs. Efforts to improve the level of costing evidence—such as by integrating cost analysis within implementation studies and trials of immunization scale up—could allow programs to better allocate resources for coverage improvement. Greater adoption of standardized cost reporting methods would also enable the synthesis and use of cost data.Electronic supplementary materialThe online version of this article (10.1186/s12913-019-4468-4) contains supplementary material, which is available to authorized users.

Policy responses to COVID-19, particularly those related to non-pharmaceutical interventions, are unprecedented in scale and scope. However, policy impact evaluations require a complex combination of circumstance, study design, data, statistics, and analysis. Beyond the issues that are faced for any policy, evaluation of COVID-19 policies is complicated by additional challenges related to infectious disease dynamics and a multiplicity of interventions. The methods needed for policy-level impact evaluation are not often used or taught in epidemiology, and differ in important ways that may not be obvious. Methodological complications of policy evaluations can make it difficult for decision-makers and researchers to synthesize and evaluate strength of evidence in COVID-19 health policy papers. We (1) introduce the basic suite of policy impact evaluation designs for observational data, including cross-sectional analyses, pre/post, interrupted time-series, and difference-in-differences analysis, (2) demonstrate key ways in which the requirements and assumptions underlying these designs are often violated in the context of COVID-19, and (3) provide decision-makers and reviewers a conceptual and graphical guide to identifying these key violations. The overall goal of this paper is to help epidemiologists, policy-makers, journal editors, journalists, researchers, and other research consumers understand and weigh the strengths and limitations of evidence.

The world is not on track to achieve the goals for immunization coverage and equity described by the World Health Organization’s Global Vaccine Action Plan. Many countries struggle to increase coverage of routine vaccination, and there is little evidence about how to do so effectively. In India in 2016, only 62% of children had received a full course of basic vaccines. In response, in 2017–18 the government implemented Intensified Mission Indradhanush (IMI), a nationwide effort to improve coverage and equity using a campaign-style strategy. Campaign-style approaches to routine vaccine delivery like IMI, sometimes called ‘periodic intensification of routine immunization’ (PIRI), are widely used, but there is little robust evidence on their effectiveness. We conducted a quasi-experimental evaluation of IMI using routine data on vaccine doses delivered, comparing districts participating and not participating in IMI. Our sample included all districts that could be merged with India’s 2016 Demographic and Health Surveys data and had available data for the full study period. We used controlled interrupted time-series analysis to estimate the impact of IMI during the 4-month implementation period and in subsequent months. This method assumes that, if IMI had not occurred, vaccination trends would have changed in the same way in the participating and not participating districts. We found that, during implementation, IMI increased delivery of 13 infant vaccines, with a median effect of 10.6% (95% confidence interval 5.1% to 16.5%). We did not find evidence of a sustained effect during the 8 months after implementation ended. Over the 12 months from the beginning of implementation, we estimated reductions in the number of under-immunized children that were large but not statistically significant, ranging from 3.9% (−6.9% to 13.7%) to 35.7% (−7.5% to 77.4%) for different vaccines. The largest effects were for the first doses of vaccines against diphtheria-tetanus-pertussis and polio: IMI reached approximately one-third of children who would otherwise not have received these vaccines. This suggests that PIRI can be successful in increasing routine immunization coverage, particularly for early infant vaccines, but other approaches may be needed for sustained coverage improvements.

Background To plan for the financial sustainability of immunization programs and make informed decisions to improve immunization coverage and equity, decision-makers need to know how much these programs cost beyond the cost of the vaccine. Non-vaccine delivery cost estimates can significantly influence the cost-effectiveness estimates used to allocate resources at the country level. However, many low- and middle-income countries (LMICs) do not have immunization delivery unit cost estimates available, or have estimates that are uncertain, unreliable, or old. We undertook a Bayesian evidence synthesis to generate country-level estimates of immunization delivery unit costs for LMICs. Methods From a database of empirical immunization costing studies, we extracted estimates of the delivery cost per dose for routine childhood immunization services, excluding vaccine costs. A Bayesian meta-regression model was used to regress delivery cost per dose estimates, stratified by cost category, against a set of predictor variables including country-level [gross domestic product per capita, reported diphtheria-tetanus-pertussis third dose coverage (DTP3), population, and number of doses in the routine vaccination schedule] and study-level (study year, single antigen or programmatic cost per dose, and financial or economic cost) predictors. The fitted prediction model was used to generate standardized estimates of the routine immunization delivery cost per dose for each LMIC for 2009–2018. Alternative regression models were specified in sensitivity analyses. Results We estimated the prediction model using the results from 29 individual studies, covering 24 countries. The predicted economic cost per dose for routine delivery of childhood vaccines (2018 US dollars), not including the price of the vaccine, was $1.87 (95% uncertainty interval $0.64–4.38) across all LMICs. By individual cost category, the programmatic economic cost per dose for routine delivery of childhood vaccines was $0.74 ($0.26–1.70) for labor, $0.26 ($0.08–0.67) for supply chain, $0.22 ($0.06–0.57) for capital, and $0.65 ($0.20–1.66) for other service delivery costs. Conclusions Accurate immunization delivery costs are necessary for assessing the cost-effectiveness and strategic planning needs of immunization programs. The cost estimates from this analysis provide a broad indication of immunization delivery costs that may be useful when accurate local data are unavailable.

BackgroundIn low- and middle-income countries, multisite costing studies are increasingly used to estimate healthcare program costs. These studies have employed a variety of estimators to summarize sample data and make inferences about overall program costs.ObjectiveWe conducted a systematic review and simulation study to describe these estimation methods and quantify their performance in terms of expected bias and variance.MethodsWe reviewed the published literature through January 2017 to identify multisite costing studies conducted in low- and middle-income countries and extracted data on analytic approaches. To assess estimator performance under realistic conditions, we conducted a simulation study based on 20 empirical cost data sets.ResultsThe most commonly used estimators were the volume-weighted mean and the simple mean, despite theoretical reasons to expect bias in the simple mean. When we tested various estimators in realistic study scenarios, the simple mean exhibited an upward bias ranging from 12% to 113% of the true cost across a range of study sample sizes and data sets. The volume-weighted mean exhibited minimal bias and substantially lower root mean squared error. Further gains were possible using estimators that incorporated auxiliary information on delivery volumes.ConclusionsThe choice of summary estimator in multisite costing studies can significantly influence study findings and, therefore, the economic analyses they inform. Use of the simple mean to summarize the results of multisite costing studies should be considered inappropriate. Our study demonstrates that several alternative better-performing methods are available.

BackgroundDespite the almost universal adoption of Integrated Management of Childhood Illness (IMCI) guidelines for the diagnosis and treatment of sick children under the age of five in low- and middle-income countries, child mortality remains high in many settings. One possible explanation of the continued high mortality burden is lack of compliance with diagnostic and treatment protocols. We test this hypothesis in a sample of children with severe illness in the Democratic Republic of the Congo (DRC).MethodsOne thousand one hundred eighty under-five clinical visits were observed across a regionally representative sample of 321 facilities in the DRC. Based on a detailed list of disease symptoms observed, patients with severe febrile disease (including malaria), severe pneumonia, and severe dehydration were identified. For all three disease categories, treatments were then compared to recommended case management following IMCI guidelines.ResultsOut of 1180 under-five consultations observed, 332 patients (28%) had signs of severe febrile disease, 189 patients (16%) had signs of severe pneumonia, and 19 patients (2%) had signs of severe dehydration. Overall, providers gave the IMCI-recommended treatment in 42% of cases of these three severe diseases. Less than 15% of children with severe disease were recommended to receive in-patient care either in the facility they visited or in a higher-level facility.ConclusionsThese results suggest that adherence to IMCI protocols for severe disease remains remarkably low in the DRC. There is a critical need to identify and implement effective approaches for improving the quality of care for severely ill children in settings with high child mortality.

IntroductionThe impact of policies on COVID-19 outcomes is one of the most important questions of our time. Unfortunately, there are substantial concerns about the strength and quality of the literature examining policy impacts. This study systematically assessed the currently published COVID-19 policy impact literature for a checklist of study design elements and methodological issues.MethodsWe included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26 or earlier and screening, all studies were reviewed by three reviewers independently and in consensus. The review tool was based on review guidance for assessing COVID-19 health policy impact evaluation analyses, including first identifying the assumptions behind the methods used, followed by assessing graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating.ResultsAfter 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside due to inappropriate study design (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, we found that only four (or by a stricter standard, only one) of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.DiscussionThe current literature directly evaluating the impact of COVID-19 policies largely fails to meet key design criteria for useful inference. This may be partially due to the circumstances for evaluation being particularly difficult, as well as a context with desire for rapid publication, the importance of the topic, and weak peer review processes. Importantly, weak evidence is non-informative and does not indicate how effective these policies were on COVID-19 outcomes.

Background In an effort to improve population health, many low- and middle-income countries (LMICs) have expanded access to public primary care facilities and removed user fees for services in these facilities. However, a growing literature suggests that many patients bypass nearby primary care facilities to seek care at more distant or higher-level facilities. Patients in urban areas, a growing segment of the population in LMICs, generally have more options for where to seek care than patients in rural areas. However, evidence on care-seeking trajectories and bypassing patterns in urban areas remains relatively scarce. Methods We obtained a complete list of public health facilities and interviewed randomly selected informal sector households across 31 urban areas in Lusaka District, Zambia. All households and facilities listed were geocoded, and care-seeking trajectories mapped across the entire urban area. We analyzed three types of bypassing: i) not using health centers or health posts for primary care; ii) seeking care outside of the residential neighborhood; iii) directly seeking care at teaching hospitals. Results A total of 620 households were interviewed, linked to 88 health facilities. Among 571 adults who had recently sought non-emergency care, 65% sought care at a hospital. Among 141 children who recently sought care for diarrhea, cough, fever, or fast breathing, 34% sought care at a hospital. 71% of adults bypassed primary care facilities, 26% bypassed health centers and hospitals close to them for more distant facilities, and 8% directly sought care at a teaching hospital. Bypassing was also observed for 59% of children, who were more likely to seek care outside of the formal care sector, with 21% of children treated at drug shops or pharmacies. Conclusions The results presented here strongly highlight the complexity of urban health systems. Most adult patients in Lusaka do not use public primary health facilities for non-emergency care, and heavily rely on pharmacies and drug shops for treatment of children. Major efforts will likely be needed if the government wants to instate health centers as the principal primary care access point in this setting.