Impact evaluation in a multi-input multi-output setting: Evidence on the effect of additional resources for schools

Abstract

This paper proposes an innovative approach to evaluate the causal impact of a policy change in a multi-input multi-output setting. It combines insights from econometric impact evaluation techniques and efficiency analysis. In particular, the current paper accounts for endogeneity issues by introducing a quasi-experimental setting within a conditional multi-input multi-output efficiency framework and by decomposing the overall efficiency between 'group-specific' efficiency (i.e., reflecting internal managerial inefficiency) and 'program' efficiency (i.e., explaining the impact of the policy intervention on performance). This framework allows the researcher to interpret the efficiency scores in terms of causality. The practical usefulness of the methodology is demonstrated through an application to secondary schools in Flanders, Belgium. By exploiting an exogenous threshold, the paper examines whether additional resources for disadvantaged students impact the efficiency of schools. The empirical results indicate that additional resources do not causally influence efficiency around the threshold.

Policy evaluation and efficiency: a systematic literature review

“…This is reflected also in the works on efficiency for policy evaluation. Among the papers included in this review only five implemented an FDH approach (Cherchye et al., 2007a; Haelermans and De Witte, 2012; Asatryan and De Witte, 2015; Cordero et al., 2017; D'Inverno et al., 2021). A possible explanation can be traced back to the fact that nonconvex models are computationally more demanding, as they require to solve binary mixed integer programming problems instead of the classical linear problems, and the solution to these problems is less likely to be covered in the commonly available software packages (Kerstens and Van de Woestyne, 2018).…”

“…(2002) and Daraio and Simar (2007) with the introduction of robust and conditional models as they allow to investigate the effect of possible environmental variables on the efficiency scores, without assuming the separability condition. Despite attractive features of the conditional efficiency approach (see De Witte and Kortelainen, 2013), they have been used merely in education (Cherchye et al., 2007a; Haelermans and De Witte, 2012; De Witte et al., 2013b; Cordero et al., 2018; D'Inverno et al., 2021), local government (Asatryan and De Witte, 2015; Cordero et al., 2017; Titl and De Witte, 2021), and innovation (Halkos and Tzeremes, 2013) applications. The absence of a wider diffusion can be partly attributed to the fact that conditional models rely on a sophisticated theoretical apparatus and partly to the fact that at the moment no statistical software provide built‐in procedure to implement conditional analysis.…”

“…The systematic literature review revealed 22 papers. Within the education field, the majority of applications focuses on higher education (Glass et al., 2006; Soares de Mello et al., 2006; Waldo, 2007; Chang et al., 2009; Coupet and Barnum, 2010; Kantabutra and Tang, 2010; Zhang et al., 2011; Montoneri et al., 2012; Sav, 2012; Tochkov et al., 2012; De Witte et al., 2013a, 2013b; Schubert and Yang, 2016; Carrington et al., 2018; Yang et al., 2018; Papadimitriou and Johnes, 2019), followed by secondary education (Haelermans and De Witte, 2012; Cordero et al., 2018; D'Inverno et al., 2021) and primary education (Cherchye et al., 2007a; Santín & Sicilia, 2018). Only Johnes and Virmani (2020) follow the students across different levels (excluding higher education).…”

“…These numbers are surprisingly low considering that none of the investigated interventions in our review considers random assignments. The empirical fields where endogeneity is more discussed are education (Waldo, 2007; Schubert and Yang, 2016; Santín & Sicilia, 2018; Papadimitriou and Johnes, 2019; D'Inverno et al., 2021), local government and environment (Halkos and Tzeremes, 2011b; Broersma et al., 2013; Asatryan and De Witte, 2015; Cordero et al., 2017; Ma et al., 2020), and environment (Pan et al., 2019; Czyźewski et al., 2020; Ma et al., 2020; Zhang et al., 2020). Interestingly, endogeneity is regarded, in any field of applications, only from a policy evaluation perspective, that is, it is discussed in terms of correlation between a regressor and the error term.…”

Local government size and service level provision. Evidence from conditional non-parametric analysis

“…Recent studies have evaluated the impact of reforms about the municipal fragmentation or amalgamation on various indicators, mostly using Difference-in-Differences techniques (for a review see for example Tavares, 2018;Swianiewicz, 2018). Following an emerging stream of literature (see for example Santín and Sicilia, 2017;D'Inverno et al, 2019), future lines of research might combine the toolbox of performance and efficiency analysis applied in this paper with policy evaluation techniques so to provide quasi-experimental evidence and therefore causal interpretation of the findings in the municipal size debate. Next, future research could explore cost and allocative efficiencies, and relate these to the service level provision.…”