Reviewing Counterfactual Analyses to Assess Impacts of EU Rural Development Programmes: What Lessons Can Be Learned from the 2007–2013 Ex-Post Evaluations?

Abstract

Counterfactual analysis has been recommended as a means of assessing the impacts of European Rural Development Programmes (RDP) over recent years, although its application has been scarce to date. This paper examines the use of counterfactual analysis to assess socioeconomic impacts in a set of 2007–2013 ex-post evaluations. The analysis undertaken shows that a wide variety of counterfactual approaches have been applied, although certain barriers still remain to address the estimation of RDP impacts following the EU evaluation standards. Furthermore, we noted that impacts provided by individual RDP evaluations may hardly be aggregated, making it difficult to draw clear conclusions about the effectiveness of rural development policy at the EU level.

Success or Waste of Taxpayer Money? Impact Assessment of Rural Development Programs in Hungary

“…Using directly comparable methodology, the authors of found negligible impacts for the programs on both Slovakian rural regions and Hungarian regions, respectively. We may additionally compare our results to the evaluations discussed in [21]. These also report minor, insignificant and sometimes negative impacts.…”

“…These also report minor, insignificant and sometimes negative impacts. However, [21] stressed that the quoted evaluations were done on "tiny samples" and "forced assumptions". None of these critiques applies for our case, due to the wealth of data provided by a six-year panel of 174 observations per year.…”

“…Thus, the main challenge of impact analysis is to construct a convincing and reasonable comparison group (counterfactual) for beneficiaries in the absence of data. The very recent paper by Castaño et al [21] explores in detail the use of counterfactuals in the ex-post evaluation of 2007-2013 RDP. Using the example of seven evaluations employing rather different empirical methodologies, the authors highlighted the difficulties with respect to data needs and limiting methodological assumptions.…”

Review of international approaches to evaluating rural and community development investment and supports

“…There is a recent body of literature attempting to answer such questions and provide empirical estimates of the impact of rural and community development interventions (see for example, Michalek et al, 2020;Bakucs et al, 2019, Castaño et al, 2019. This literature highlights the substantial challenges that exist for academic researchers and policy evaluators alike, due to the highly complex and cross-sectoral nature of rural and urban communities and the multiplicity of outcomes associated with supports.…”

“…Counterfactual analysis has been recommended as a means of assessing the impacts of Rural Development Programmes (RDP) in Europe over recent years, although its application has been scarce to date (Castaño et al, 2019). However, the assessment of the impact of any rural development programme is often driven by data availability, the type of the rural development measure to be analysed, the geographical coverage and the econometric approach (Michalek et al, 2020).…”