Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments

Shadish, William R.; Clark, M. H.; Steiner, Péter

doi:10.1198/016214508000000733

Cited by 423 publications

(469 citation statements)

References 44 publications

Supporting

Mentioning

453

Contrasting

Unclassified

Order By: Relevance

“…Critics may contend that they will lead to comparison studies that can only be done in laboratory settings (e.g., Shadish, Clark & Steiner, 2008). However, we have reviewed recent field-study comparisons of this type (Cook et al, 2008) and found a number that took place in field settings and still were able to remedy many of the problems in PMO (e.g., Aiken, West, Schwalm, Carroll & Hsuing, 1998;Diaz & Handa, 2006).…”

Section: Discussionmentioning

confidence: 99%

A Case Study About Why It Can Be Difficult To Test Whether Propensity Score Analysis Works in Field Experiments

Shadish

Steiner

Cook³

2013

JMMSS

Self Cite

View full text Add to dashboard Cite

Peikes, Moreno and Orzol (2008) sensibly caution researchers that propensity score analysis may not lead to valid causal inference in field applications. But at the same time, they made the far stronger claim to have performed an ideal test of whether propensity score matching in quasi-experimental data is capable of approximating the results of a randomized experiment in their dataset, and that this ideal test showed that such matching could not do so. In this article we show that their study does not support that conclusion because it failed to meet a number of basic criteria for an ideal test. By implication, many other purported tests of the effectiveness of propensity score analysis probably also fail to meet these criteria, and are therefore questionable contributions to the literature on the effects of propensity score analysis. Keywords: Propensity Scores, Strong Ignorability, Quasi-Experiments, Within-Study ComparisonIn 2008, Peikes, Moreno and Orzol (henceforth PMO) published a case study in The American Statistician that cautioned social program evaluators that propensity score analysis may yield quite different results than those from a randomized experiment. We join them in that caution because we doubt that many applications of propensity scores meet some of the most basic conditions for their valid use (Shadish, 2012;Steiner, Cook & Shadish, 2011;Shadish, Cook, Steiner & Clark, 2010). In that sense, we applaud the PMO article. Field researchers need to appreciate how difficult it can be to use propensity score analysis in a way that yields confidence in the results.At the same time, however, PMO made a second claim, that they performed an "ideal" (pp. 222, 223, 230) test of whether propensity score matching in quasi-experimental data could approximate the results of a randomized experiment. In fact, a careful analysis of PMO suggests just the opposite conclusion, that they neither implemented an ideal propensity score analysis nor an ideal comparison of results from a propensity score analysis to results from a randomized experiment. In this article, we show why the PMO study was not ideal in both respects. Just as practitioners need to appreciate how difficult it can be to use propensity

show abstract

Section: Discussionmentioning

confidence: 99%

A Case Study About Why It Can Be Difficult To Test Whether Propensity Score Analysis Works in Field Experiments

Shadish

Steiner

Cook³

2013

JMMSS

Self Cite

View full text Add to dashboard Cite

show abstract

“…In another recent study, Shadish et al (2008) conducted an intriguing experiment by randomly assigning one group of individuals to be randomly assigned to treatment status and the other to self-select one of the two treatment options (mathematics or vocabulary training). The authors found that propensity score matching greatly reduced the bias of impact estimates when the full set of available covariates was used, including pretests, but did poorly when only predictors of convenience (sex, age, marital status, and ethnicity) were used.…”

Section: Can Nonexperimental Methods Replicate Experimental Findings?mentioning

confidence: 99%

Setting up social experiments: the good, the bad, and the ugly

Barnow

2010

ZAF

View full text Add to dashboard Cite

Standard-Nutzungsbedingungen:Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden.Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen.Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Abstract It is widely agreed that randomized controlled trials -social experiments -are the gold standard for evaluating social programs. There are, however, many important issues that cannot be tested using social experiments, and often things go wrong when conducting social experiments. This paper explores these issues and offers suggestions on ways to deal with commonly encountered problems. Social experiments are preferred because random assignment assures that any differences between the treatment and control groups are due to the intervention and not some other factor; also, the results of social experiments are more easily explained and accepted by policy officials. Experimental evaluations often lack external validity and cannot control for entry effects, scale and general equilibrium effects, and aspects of the intervention that were not randomly assigned. Experiments can also lead to biased impact estimates if the control group changes its behavior or if changing the number selected changes the impact. Other problems with conducting social experiments include increased time and cost, and legal and ethical issues related to excluding people from the treatment. Things that sometimes go wrong in social experiments include programs cheating on random assignment, and participants and/or staff not understanding the intervention rules. The random assignment evaluation of the Job Training Partnership Act in the United States is used as a case study to illustrate the issues. Terms of use: Documents in

show abstract

“…The authors wish to thank Daniel Rodriguez for suggesting the Cook, Shadish, and Wong (2008) and the Shadish, Clark, and Steiner (2008) references. We also benefited from comments by participants in the Conference on Low-Carbon Cities: Land Use and Transportation Intervention, Xi'an, China, June 12-13, 2014, and the World Symposium on Transport and Land Use Research, Delft, the Netherlands, June 24-27, 2014.…”

Section: Acknowledgementsmentioning

confidence: 99%

“…However, if either BEeff or ATeff is near 0 in reality, its estimate may lie just on the "other side" of 0 (compared to the sign of the larger effect) through random variation, which could lead to an estimated BEP slightly outside [0,1]. 3 The evaluation literature includes a number of comparisons (e.g., Shadish, Clark, and Steiner 2008;Cook, Shadish, and Wong 2008) of treatment effects (TE) estimated from randomized experiments (RE, considered the gold standard), TE RE , with those estimated from observational studies (OS), TE OS -most rigorously using the same sample for both measurements. In such cases it is common to report bias reductions for various methods that control for selection bias.…”

mentioning

confidence: 99%

Viewpoint: Quantifying residential self-selection effects: A review of methods and findings from applications of propensity score and sample selection approaches

Mokhtarian

Herick

2016

JTLU

View full text Add to dashboard Cite

Abstract:The phenomenon whereby individuals self-select into their residential environment based on previously determined preferences for how to travel is known as residential self-selection (RSS). Numerous studies have investigated the influence of RSS on the estimated effect of the built environment on travel behavior. However, surprisingly few have actually quantified its effect in terms of partitioning the total influence of the built environment (BE) on travel behavior into a component attributable to RSS and one attributable to the built environment itself. This paper reviews 10 analyses (found in seven studies) that have quantified the proportion of the total influence of the built environment that is due to the BE itself (which we call the BEP), using either propensity-score or sample-selection approaches to control for RSS. After first outlining the basics of each approach, we then explain the various methods used to compute the BEP, followed by a discussion of the empirical results. The estimated BEPs vary widely, ranging from 34 percent to 98 percent. A number of reasons for these disparities are suggested, but there is considerable divergence in estimates even when many of these factors are held constant. Additional research is called for to better understand the circumstances under which the BEP is higher or lower. Keywords:Residential self-selection, propensity score, sample selection, travel behavior IntroductionThe influence of the built environment (BE) on travel behavior (TB) is of considerable interest to transportation policy and planning, particularly since it is the most obvious limiting factor on whether individuals even have opportunities to make certain decisions with respect to their behavior. Without tracks, there is no rail transportation; without roads, there is no driving; and increased incidence of 10JOURNAL OF TRANSPORT AND LAND USE 9.1 aesthetic walking paths leading to interesting destinations will surely encourage, if not lead to, more walking in an area. It remains an open question, however, how much the built environment itself brings about a certain kind of behavior versus how much one's proclivities for a certain kind of behavior influence one's choice of built environment. To the extent the latter is true, policies that attempt to induce more sustainable travel behavior by shaping the built environment could fall short of expectations if large numbers of people end up living in sustainable built environments (perhaps due to financial incentives or a sizable increase in the supply of such environments) without having the proclivity to travel more sustainably. The phenomenon whereby individuals self-select into their residential environment based on previously determined preferences for how to travel is known as residential self-selection (RSS). In the past two decades, numerous studies have analyzed the influence of the built environment on travel behavior after controlling for self-selection. In the process, a variety of approaches have been employed: direct questioning, ...

show abstract

Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments

Cited by 423 publications

References 44 publications

A Case Study About Why It Can Be Difficult To Test Whether Propensity Score Analysis Works in Field Experiments

A Case Study About Why It Can Be Difficult To Test Whether Propensity Score Analysis Works in Field Experiments

Setting up social experiments: the good, the bad, and the ugly

Viewpoint: Quantifying residential self-selection effects: A review of methods and findings from applications of propensity score and sample selection approaches

Contact Info

Product

Resources

About