Multiple Randomization Designs

Bajari, Patrick; Burdick, Brian; Imbens, Guido W.; Masoero, Lorenzo; McQueen, James M.; Richardson, Thomas; Rosen, Ido M.

doi:10.48550/arxiv.2112.13495

Cited by 9 publications

(18 citation statements)

References 13 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We view this paper as orthogonal to this literature, but will eventually compare against a recent state-of-the-art design, so-called two-sided randomization [29,5], that is specific to the context of two-sided marketplaces (e.g. the one we simulate).…”

Section: Related Literaturementioning

confidence: 99%

Markovian Interference in Experiments

Farias¹,

Li²,

Peng³

et al. 2022

Preprint

View full text Add to dashboard Cite

We consider experiments in dynamical systems where interventions on some experimental units impact other units through a limiting constraint (such as a limited inventory). Despite outsize practical importance, the best estimators for this 'Markovian' interference problem are largely heuristic in nature, and their bias is not well understood. We formalize the problem of inference in such experiments as one of policy evaluation. Off-policy estimators, while unbiased, apparently incur a large penalty in variance relative to state-of-the-art heuristics. We introduce an on-policy estimator: the Differences-In-Q's (DQ) estimator. We show that the DQ estimator can in general have exponentially smaller variance than off-policy evaluation. At the same time, its bias is second order in the impact of the intervention. This yields a striking bias-variance tradeoff so that the DQ estimator effectively dominates state-of-the-art alternatives. From a theoretical perspective, we introduce three separate novel techniques that are of independent interest in the theory of Reinforcement Learning (RL). Our empirical evaluation includes a set of experiments on a city-scale ride-hailing simulator.

show abstract

Section: Related Literaturementioning

confidence: 99%

Markovian Interference in Experiments

Farias¹,

Li²,

Peng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Our bounds and estimates of the treatment effect can be used to characterize various heterogeneous spillover effects. The kinds of spillover effects that are identified depends on the experimental design, see Bajari et al (2021). The following illustrative example is based on their Section 6.…”

Section: Interferencementioning

confidence: 99%

“…In the first group both the buyers and the sellers are treated, in the second group the buyers and not the sellers are treated, in the third group the sellers and not the buyers are treated, and in the fourth group neither the sellers nor the buyers are treated. Bajari et al (2021) define the average buyer (seller) spillover effect to be the average difference in the outcomes in the second (third) and fourth groups. Following the symmetrization argument of Section 5.1, the arguments of Section 3 can be used to identify and estimate the distribution of such spillover effects.…”

Section: Interferencementioning

confidence: 99%

See 1 more Smart Citation

Heterogeneous Treatment Effects for Networks, Panels, and other Outcome Matrices

Auerbach¹,

Cai²

2022

Preprint

View full text Add to dashboard Cite

We are interested in the distribution of treatment effects for an experiment where units are randomized to treatment but outcomes are measured for pairs of units. For example, we might measure risk sharing links between households enrolled in a microfinance program, employment relationships between workers and firms exposed to a trade shock, or bids from bidders to items assigned to an auction format. Such a double randomized experimental design may be appropriate when there are social interactions, market externalities, or other spillovers across units assigned to the same treatment. Or it may describe a natural or quasi experiment given to the researcher. In this paper, we propose a new empirical strategy based on comparing the eigenvalues of the outcome matrices associated with each treatment. Our proposal is based on a new matrix analog of the Fréchet-Hoeffding bounds that play a key role in the standard theory. We first use this result to bound the distribution of treatment effects. We then propose a new matrix analog of quantile treatment effects based on the difference in the eigenvalues. We call this analog spectral treatment effects.

show abstract

“…Moreover, a proper understanding of the interference issue in relation to causal inference directly impacts engineering of more purposeful interventions and design of more effective A/B testing for ad placement. Alternatively, randomized experiments via bipartite graphs offer a useful formalism to study two-sided market experiments under violation of iid assumption (Pouget-Abadie et al, 2018 , 2019 ; Bajari et al, 2021 ; Harshaw et al, 2021 ; Johari et al, 2022 ). This stands in contrast with interference that occurs on networks where all units are of the same type (e.g., ads in a block)—in bipartite experiments, there is a distinction between units that can be subject to an intervention and units whose responses are of interest to the experimenter.…”

Section: Introductionmentioning

confidence: 99%

Causal Inference in the Presence of Interference in Sponsored Search Advertising

et al. 2022

View full text Add to dashboard Cite

In classical causal inference, inferring cause-effect relations from data relies on the assumption that units are independent and identically distributed. This assumption is violated in settings where units are related through a network of dependencies. An example of such a setting is ad placement in sponsored search advertising, where the likelihood of a user clicking on a particular ad is potentially influenced by where it is placed and where other ads are placed on the search result page. In such scenarios, confounding arises due to not only the individual ad-level covariates but also the placements and covariates of other ads in the system. In this paper, we leverage the language of causal inference in the presence of interference to model interactions among the ads. Quantification of such interactions allows us to better understand the click behavior of users, which in turn impacts the revenue of the host search engine and enhances user satisfaction. We illustrate the utility of our formalization through experiments carried out on the ad placement system of the Bing search engine.

show abstract

Multiple Randomization Designs

Cited by 9 publications

References 13 publications

Markovian Interference in Experiments

Markovian Interference in Experiments

Heterogeneous Treatment Effects for Networks, Panels, and other Outcome Matrices

Causal Inference in the Presence of Interference in Sponsored Search Advertising

Contact Info

Product

Resources

About