Do studies evaluating early-life policy interventions fully adhere to the critical conditions of difference-in-differences? A systematic review

Abstract

ObjectivesTo assess the reporting and methodological quality of early-life policy intervention papers that applied difference-in-differences (DiD) analysis.Study designSystematic review.Data sourcesPapers applying DiD of early-life policy interventions in high-income countries as identified by searching Medline, Embase and Scopus databases up to December, 2022.Study eligibility criteria, participants and interventionsStudies evaluating policy interventions targeting expectant mothers, infants or children up to two years old and conducted in high income countries were included. We focused on seven critical conditions of DiD as proposed in a comprehensive checklist: data requirements, parallel trends, no-anticipation, standard statistical assumptions, common shocks, group composition and spillover.ResultsThe DiD included studies (n=19) evaluating early-life policy interventions in childhood development (n=4), healthcare utilisation and providers (n=4), nutrition programmes (n=3) and economic policies such as prenatal care expansion (n=8). Although none of the included studies met all critical conditions, the most reported and adhered to critical conditions were data requirements (n=18), standard statistical assumptions (n=11) and the parallel trends assumption (n=9). No-anticipation and spillover were explicitly reported and adhered to in two studies and one study, respectively.ConclusionsThis review highlights current deficiencies in the reporting and methodological quality of studies using DiD to evaluate early-life policy interventions. As the validity of study conclusions and consequent implications for policy depend on the extent to which critical conditions are met, this shortcoming is concerning. We recommend that researchers use the described checklist to improve the transparency and validity of their evaluations. The checklist should be further refined by adding order of importance or knock-out criteria and may also help facilitate uniform terminology. This will hopefully encourage reliable DiD evaluations and thus contribute to better policies relating to expectant mothers, infants and children.