The Power of Optimization Over Randomization in Designing Experiments Involving Small Samples

Bertsimas, Dimitris; Johnson, Martin P.; Kallus, Nathan

doi:10.1287/opre.2015.1361

Cited by 63 publications

(63 citation statements)

References 18 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fixing ρ >0 and endowing

F = span {1, x_{1}, \dots, x_{d}, x_{1}^{2} / ρ, \dots, x_{d}^{2} / ρ, x_{1} x_{2} / false(2 italicρ false), \dots, x_{d - 1} x_{d} / false(2 italicρ false)}

with the ∞‐norm and normalizing the data will recover the method of Bertsimas et al . (), which optimizes the balance in covariate means and centred second moments by using mixed integer programming.…”

Section: The Effect Of Structural Information and Lack Thereofmentioning

confidence: 99%

“…This both generalizes and formalizes an observation that was made by Bertsimas et al . (). We develop algorithms for computing optimal designs by using mixed integer programming and semidefinite programming and provide hypothesis tests for these designs.…”

Section: Introductionmentioning

confidence: 97%

“…Bertsimas et al . () defined imbalance as the sum of discrepancies in group means and variances and found optimal designs with globally minimal imbalance.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

OptimalA PrioriBalance in the Design of Controlled Experiments

Kallus

2017

Journal of the Royal Statistical Society Series B: Statistical Methodology

Self Cite

111

View full text Add to dashboard Cite

Summary. We develop a unified theory of designs for controlled experiments that balance baseline covariates a priori (before treatment and before randomization) using the framework of minimax variance and a new method called kernel allocation. We show that any notion of a priori balance must go hand in hand with a notion of structure, since with no structure on the dependence of outcomes on baseline covariates complete randomization (no special covariate balance) is always minimax optimal. Restricting the structure of dependence, either parametrically or non-parametrically, gives rise to certain covariate imbalance metrics and optimal designs. This recovers many popular imbalance metrics and designs previously developed ad hoc, including randomized block designs, pairwise-matched allocation and rerandomization. We develop a new design method called kernel allocation based on the optimal design when structure is expressed by using kernels, which can be parametric or non-parametric. Relying on modern optimization methods, kernel allocation, which ensures nearly perfect covariate balance without biasing estimates under model misspecification, offers sizable advantages in precision and power as demonstrated in a range of real and synthetic examples. We provide strong theoretical guarantees on variance, consistency and rates of convergence and develop special algorithms for design and hypothesis testing.

show abstract

“…Fixing ρ >0 and endowing

F = span {1, x_{1}, \dots, x_{d}, x_{1}^{2} / ρ, \dots, x_{d}^{2} / ρ, x_{1} x_{2} / false(2 italicρ false), \dots, x_{d - 1} x_{d} / false(2 italicρ false)}

Section: The Effect Of Structural Information and Lack Thereofmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 97%

See 1 more Smart Citation

OptimalA PrioriBalance in the Design of Controlled Experiments

Kallus

2017

Journal of the Royal Statistical Society Series B: Statistical Methodology

Self Cite

111

View full text Add to dashboard Cite

show abstract

“…Utilizing baseline information to an even greater degree than in the stratification case, some researchers have recently suggested relying on optimization instead of randomization for assigning treatment status to subjects. Bertsimas, Johnson, and Kallus () propose a method based on discrete linear optimization, such that assignment is chosen to minimize the discrepancy between treatment groups in terms of means and variances of covariates. Kasy () considers the experiment as a statistical decision problem where the goal is to find the unique treatment assignment that minimizes a Bayesian or minimax risk function (based on the mean squared error of a point estimator).…”

Section: Dozen Thingsmentioning

confidence: 99%

The Dozen Things Experimental Economists Should Do (More of)

Czibor¹,

Jimenez-Gomez²,

List

2019

Southern Economic Journal

View full text Add to dashboard Cite

What was once broadly viewed as an impossibility—learning from experimental data in economics—has now become commonplace. Governmental bodies, think tanks, and corporations around the world employ teams of experimental researchers to answer their most pressing questions. For their part, in the past two decades academics have begun to more actively partner with organizations to generate data via field experimentation. Although this revolution in evidence‐based approaches has served to deepen the economic science, recently a credibility crisis has caused even the most ardent experimental proponents to pause. This study takes a step back from the burgeoning experimental literature and introduces 12 actions that might help to alleviate this credibility crisis and raise experimental economics to an even higher level. In this way, we view our “12 action wish list” as discussion points to enrich the field.

show abstract

“…They suggested developing plans that are robust to time effects in order to obtain null covariance between variables and time. Bertsimas et al (2015) also showed the benefits of the systematic approach in experiments with "guinea pigs", by applying mathematical optimization techniques. Ganju & Lucas (2004) studied the systematic sequencing of the experiments' order, and they indicated the inadequacy of the randomization as a practice to be followed in any situation.…”

Section: Two-level Factorial Experimental Designs In the Presence Of mentioning

confidence: 99%

Sequenciamento sistemático de experimentos fatoriais como alternativa à ordem aleatória

2017

View full text Add to dashboard Cite

Resumo: Este trabalho tem como objetivo principal discutir o uso de métodos sistemáticos para geração de designs de experimentos com boas propriedades estatísticas e custos baixos. O foco da pesquisa é o sequenciamento dos Abstract:The current study aims to discuss the use of systematic methods to generate experimental designs with good statistical properties and low costs. The research focuses on the sequence of experiments and on analysis the results of three different approaches used to build (orthogonal and non-orthogonal) two-level factorial designs, wherein sequencing is randomly or systematically performed. The study simulated the design generated by each approach in the context of an actual glass container manufacturing process, with and without the presence of linear trend effects. The results indicate that, in comparison to the random order, systematic sequences may lead to fewer factor level changes and to increased robustness to linear trend effects. Therefore, they may attach design cost and quality.

show abstract

The Power of Optimization Over Randomization in Designing Experiments Involving Small Samples

Cited by 63 publications

References 18 publications

OptimalA PrioriBalance in the Design of Controlled Experiments

OptimalA PrioriBalance in the Design of Controlled Experiments

The Dozen Things Experimental Economists Should Do (More of)

Sequenciamento sistemático de experimentos fatoriais como alternativa à ordem aleatória

Contact Info

Product

Resources

About