Dealing with Design Failures in Randomized Field Experiments: Analytic Issues Regarding the Evaluation of Treatment Effects

Abstract

With an increasing number of criminal justice scholars conducting randomized field experiments, there are several analytic issues related to such studies that our discipline must begin to address more systematically. For example, treatment dilution and treatment migration are common forms of randomization implementation failure in field experiments, and a review of the criminological literature on experiments reveals a lack of consensus as to how these problems should be handled when evaluating treatment effects. In addition, issues related to statistical power and desired sample size remain unresolved. Given the relatively longer history of dealing with these issues in medicine, literature from that field is reviewed, providing additional insights regarding the dilemmas created by various design failures in randomized field experiments.

Lessons learned from the implementation of two randomized trials in a criminal court setting

“…In this regard, our decision to hold all of the relevant events within a single, structured court appearance (the initial allocution, the randomization, and the imposition of a specific sentence) proved to be effective, as no one ever attempted to withdraw the applicable plea or plea offer post-randomization in either site. Our experience mirrors suggestions drawn from other randomized experiments in criminal justice setting (e.g., Berk et al 1988;Feder et al 2000;Gartin 1995), and we believe that the approach ultimately taken may serve as a model for other randomized trials in court settings.…”

“…Conversely, we detected no evidence that any of the offenders who were not randomly assigned to a program somehow attended one anyway. In short, our intent-to-treat protocols virtually mirrored the factual interventions that the offenders did or did not experience, a critical element in any randomized trial (e.g., see Angrist 2006;Gartin 1995).…”

Instrumental variables methods in experimental criminological research: what, why and how

“…A major concern in the criminological literature is the possibility of a failed research design (see, e.g., Farrington 1983;Rezmovic et al 1981;Gartin 1995). Gartin (1995) notes that two sorts of design failure seem especially likely.…”

Instrumental Variables Methods in Experimental Criminological Research: What, Why, and How?

“…In practice, non-compliance is usually avoidable in research using human subjects. Gartin (1995) discusses criminological examples; non-compliance has long been discussed in randomized clinical trials (see, e.g., Efron and Feldman, 1991).…”

“…A major concern in the criminological literature is the possibility of a failed research design (see, e.g., Farrington, 1983, Rezmovic, et al, 1981, and Gartin, 1995. Gartin (1995) notes that two sorts of design failure seem especially likely.…”