Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts

Abstract

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.. Oxford University Press and The Review of Economic Studies, Ltd. are collaborating with JSTOR to digitize, preserve and extend access to The Review of Economic Studies.The conventional approach to social programme evaluation focuses on estimating mean impacts of programmes. Yet many interesting questions regarding the political economy of programmes, the distribution of programme benefits and the option values conferred on programme participants require knowledge of the distribution of impacts, or features of it. This paper presents evidence that heterogeneity in response to programmes is empirically important and that classical probability inequalities are not very informative in producing estimates or bounds on the distribution of programme impacts. We explore two methods for supplementing the information in these inequalities based on assumptions about participant decision-making processes and about the strength in dependence between outcomes in the participation and non-participation states. Dependence is produced as a consequence of rational choice by participants. We test for stochastic rationality among programme participants and present and implement methods for estimating the option values of social programmes.And the Lord said, Because the cry of Sodom and Gomorrah is great, and because their sin is very grievous; I will go down now, and see whether they have done altogether according to the cry of it, which is come unto me; and if not, I will know.. . And Abraham drew near and said, Wilt thou also destroy the righteous with the wicked? Peradventure there be fifty righteous within the city; wilt thou also destroy and not spare the place for the fifty righteous that are therein? ... And the Lord said, If Ifind in Sodom fifty righteous within the city, then I will spare all the place for their sakes. Genesis 18: Verses 20-26, King James Version 487 This content downloaded from 128.235.251.160 on Mon, 15 Dec 2014 02:47:13 AM All use subject to JSTOR Terms and Conditions 488 REVIEW OF ECONOMIC STUDIES 1. INTRODUCTION Most evaluations of social programmes focus exclusively on mean impacts. Yet, as the passage from Genesis reveals, features other than the mean are often of interest. Just as Abraham convinced the Lord to spare Sodom and Gomorrah if he could find 50 righteous persons living there, so many persons would judge programmes to be successful if enough persons, or enough of the right kinds of persons, reaped benefits from them even if the average participant did not.The case for using the mean impact to evaluate a programme rests on two key assumptions: (a) that increases in total output increase welfare; and (b) that undesirable distributional aspects of programmes a...

Mercury advisories: Information, education, and fish consumption

“…The QTE is the difference between these two expenditure shares; it measures the advisory "treatment". Note that the QTE does not identify individual impacts because it does not measure changes by those specific individuals initially at a given quantile, an effect which would require perfect rank preservation (Heckman et al (1997)). Rather a QTE reflects a shift in the overall consumption pattern across individuals, an impact of considerable importance from a public policy perspective.…”

Using Innovation Survey Data to Evaluate R&D Policy: The Case of Belgium

“…The matching relies on the intuitively attracting idea to balance the sample of program participants and comparable non-participants. Remaining differences in the outcome variable between both groups are then attributed to the treatment (Heckman et al, 1997).…”

“…The literature on the econometrics of evaluation offers different estimation strategies to correct for selection bias (see Heckman et al, 1997, Heckman et al 1999 for a survey) including the differencein-difference estimator, control function approaches (selection models), IV estimation and nonparametric matching. The difference-in-difference method requires panel data with observations before and after/while the treatment (change of subsidy status).…”

Inference on Quantile Regression Process, An Alternative

“…Abadie (2002), Buchinsky (1994), Heckman and Smith (1997), Gutenbrunner and Jureckova (1992), McFadden (1989), Koenker and Xiao (2002a), and Portnoy (2001). Just like in the classical p-sample theory, e.g.…”

“…or, for example, a stochastic dominance effect, cf. Abadie (2002), Heckman and Smith (1997), and McFadden (1989 As in Koenker and Xiao (2002a), we consider the following null hypothesis: #(t)/3"(t)-7-(t) = *(t). reT, (1) where R{t) denotes a q x p matrix, q < p = dim(/3), r£l', and \£(t) denotes a known function^: T -» R 9 .…”