Experimentation at Scale

Muralidharan, Karthik; Niehaus, Paul

doi:10.1257/jep.31.4.103

Cited by 146 publications

(91 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such spillovers can also occur within treatment or control groups, positively or negatively magnifying effects. In addition to within-and between-treatment spillover effects, there is also the possibility of spillovers from the treated group to people who are not even participating in the experiment, i.e., people beyond the control group (the interested reader should see and Muralidharan and Niehaus (2017)).…”

Section: But Did It?mentioning

confidence: 99%

How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling

et al. 2019

View full text Add to dashboard Cite

Policymakers are increasingly turning to insights gained from the experimental method as a means to inform large scale public policies. Critics view this increased usage as premature, pointing to the fact that many experimentally-tested programs fail to deliver their promise at scale. Under this view, the experimental approach drives too much public policy. Yet, if policymakers could be more confident that the original research findings would be delivered at scale, even the staunchest critics would carve out a larger role for experiments to inform policy. Leveraging the economic framework of Al-Ubaydli et al. (2019), we put forward 12 simple proposals, spanning researchers, policymakers, funders, and stakeholders, which together tackle the most vexing scalability threats. The framework highlights that only after we deepen our understanding of the scale up problem will we be on solid ground to argue that scientific experiments should hold a more prominent place in the policymaker's quiver.

show abstract

Section: But Did It?mentioning

confidence: 99%

How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Examples of such settings include large-scale experiments (see Muralidharan and Niehaus (2017)), settings where the cost of data acquisition motivates the use of random samples (see, e.g., Keels, Duncan, DeLuca, Mendenhall, and Rosenbaum (2005)), as well as analyses based on public-use census samples, like the 2010 Integrated Public Use Microdata Series (IPUMS) data (which is a 10 percent sample of the U.S. Census). Examples of such settings include large-scale experiments (see Muralidharan and Niehaus (2017)), settings where the cost of data acquisition motivates the use of random samples (see, e.g., Keels, Duncan, DeLuca, Mendenhall, and Rosenbaum (2005)), as well as analyses based on public-use census samples, like the 2010 Integrated Public Use Microdata Series (IPUMS) data (which is a 10 percent sample of the U.S. Census).…”

Section: Introductionmentioning

confidence: 99%

Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis

Abadie

Athey

Imbens

et al. 2020

ECTA

215

138

View full text Add to dashboard Cite

Consider a researcher estimating the parameters of a regression function based on data for all 50 states in the United States or on data for all visits to a website. What is the interpretation of the estimated parameters and the standard errors? In practice, researchers typically assume that the sample is randomly drawn from a large population of interest and report standard errors that are designed to capture sampling variation. This is common even in applications where it is difficult to articulate what that population of interest is, and how it differs from the sample. In this article, we explore an alternative approach to inference, which is partly design-based. In a design-based setting, the values of some of the regressors can be manipulated, perhaps through a policy intervention. Design-based uncertainty emanates from lack of knowledge about the values that the regression outcome would have taken under alternative interventions. We derive standard errors that account for design-based uncertainty instead of, or in addition to, sampling-based uncertainty. We show that our standard errors in general are smaller than the usual infinite-population sampling-based standard errors and provide conditions under which they coincide.

show abstract

“…Crucially, the PSP increases substantially if the initial positive finding is followed by at least two successful replications (section 3.5). Moreover, successful replications in different contexts are valuable for ensuring generalizability (Duflo ; Muralidharan and Niehaus , see section 3.1). Replication may also be used to measure average within‐ and across‐study variation in outcomes: when these are close, the concern about across‐context generalizability is reduced (Vivalt ).…”

Section: Dozen Thingsmentioning

confidence: 99%

“…Researchers should also consider whether results from their program are likely to generalize, being especially sensitive to heterogeneity across populations and contexts (section 3.1), and choose the optimal experiment type in light of their scaling goals (an issue we discussed in section 3.3). In terms of the population, an approach that is likely to result in better generalization is to consider the (large) population of interest first, take a representative (smaller) sample of observations this population, and then randomize those to treatment or control (Muralidharan and Niehaus ) . Compliance to the program must also be taken into account.…”

Section: Dozen Thingsmentioning

confidence: 99%

See 1 more Smart Citation

The Dozen Things Experimental Economists Should Do (More of)

Czibor¹,

Jimenez-Gomez²,

List

2019

Southern Economic Journal

View full text Add to dashboard Cite

What was once broadly viewed as an impossibility—learning from experimental data in economics—has now become commonplace. Governmental bodies, think tanks, and corporations around the world employ teams of experimental researchers to answer their most pressing questions. For their part, in the past two decades academics have begun to more actively partner with organizations to generate data via field experimentation. Although this revolution in evidence‐based approaches has served to deepen the economic science, recently a credibility crisis has caused even the most ardent experimental proponents to pause. This study takes a step back from the burgeoning experimental literature and introduces 12 actions that might help to alleviate this credibility crisis and raise experimental economics to an even higher level. In this way, we view our “12 action wish list” as discussion points to enrich the field.

show abstract

Experimentation at Scale

Cited by 146 publications

References 32 publications

How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling

How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling

Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis

The Dozen Things Experimental Economists Should Do (More of)

Contact Info

Product

Resources

About