Experimental Design in Two-Sided Platforms: An Analysis of Bias

Johari, Ramesh; Li, Hannah; Liskovich, Inessa; Weintraub, Gabriel Y.

doi:10.1287/mnsc.2021.4247

Cited by 32 publications

(32 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[Basse et al, 2016, Liu et al, 2020 compare supply-side randomization to two-sided randomization as well as to budget-split designs, showing bias can be reduced in the context of certain ad auction experiments. More recently, [Johari et al, 2020] characterizes which randomization scheme (supply-side, demand-side, or two-sided) leads to reduced bias as a function of market balance.…”

Section: Related Workmentioning

confidence: 99%

Design and Analysis of Bipartite Experiments under a Linear Exposure-Response Model

Harshaw¹,

Sävje²,

Eisenstat³

et al. 2021

Preprint

View full text Add to dashboard Cite

The bipartite experimental framework is a recently proposed causal setting, where a bipartite graph links two distinct types of units: units that receive treatment and units whose outcomes are of interest to the experimenter. Often motivated by market experiments, the bipartite experimental framework has been used for example to investigate the causal effects of supply-side changes on demand-side behavior. Similar to settings with interference and other violations of the stable unit treatment value assumption (SUTVA), additional assumptions on potential outcomes must be made for valid inference. In this paper, we consider the problem of estimating the average treatment effect in the bipartite experimental framework under a linear exposure-response model. We propose the Exposure Reweighted Linear (ERL) Estimator, an unbiased linear estimator of the average treatment effect in this setting. Furthermore, we present Exposure-Design, a cluster-based design which aims to increase the precision of the ERL estimator by realizing desirable exposure distributions. Finally, we demonstrate the effectiveness of the proposed estimator and design on a publicly available Amazon user-item review graph.

show abstract

Section: Related Workmentioning

confidence: 99%

Design and Analysis of Bipartite Experiments under a Linear Exposure-Response Model

Harshaw¹,

Sävje²,

Eisenstat³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Our setting is related to that of Bajari et al (2021); Johari et al (2022). Specifically, our double randomized experiment is a special case of their simple multiple randomization design, although these authors do not consider the distribution of treatment effects.…”

Section: Introductionmentioning

confidence: 99%

Heterogeneous Treatment Effects for Networks, Panels, and other Outcome Matrices

Auerbach¹,

Cai²

2022

Preprint

View full text Add to dashboard Cite

We are interested in the distribution of treatment effects for an experiment where units are randomized to treatment but outcomes are measured for pairs of units. For example, we might measure risk sharing links between households enrolled in a microfinance program, employment relationships between workers and firms exposed to a trade shock, or bids from bidders to items assigned to an auction format. Such a double randomized experimental design may be appropriate when there are social interactions, market externalities, or other spillovers across units assigned to the same treatment. Or it may describe a natural or quasi experiment given to the researcher. In this paper, we propose a new empirical strategy based on comparing the eigenvalues of the outcome matrices associated with each treatment. Our proposal is based on a new matrix analog of the Fréchet-Hoeffding bounds that play a key role in the standard theory. We first use this result to bound the distribution of treatment effects. We then propose a new matrix analog of quantile treatment effects based on the difference in the eigenvalues. We call this analog spectral treatment effects.

show abstract

“…Moreover, a proper understanding of the interference issue in relation to causal inference directly impacts engineering of more purposeful interventions and design of more effective A/B testing for ad placement. Alternatively, randomized experiments via bipartite graphs offer a useful formalism to study two-sided market experiments under violation of iid assumption (Pouget-Abadie et al, 2018 , 2019 ; Bajari et al, 2021 ; Harshaw et al, 2021 ; Johari et al, 2022 ). This stands in contrast with interference that occurs on networks where all units are of the same type (e.g., ads in a block)—in bipartite experiments, there is a distinction between units that can be subject to an intervention and units whose responses are of interest to the experimenter.…”

Section: Introductionmentioning

confidence: 99%

Causal Inference in the Presence of Interference in Sponsored Search Advertising

et al. 2022

View full text Add to dashboard Cite

In classical causal inference, inferring cause-effect relations from data relies on the assumption that units are independent and identically distributed. This assumption is violated in settings where units are related through a network of dependencies. An example of such a setting is ad placement in sponsored search advertising, where the likelihood of a user clicking on a particular ad is potentially influenced by where it is placed and where other ads are placed on the search result page. In such scenarios, confounding arises due to not only the individual ad-level covariates but also the placements and covariates of other ads in the system. In this paper, we leverage the language of causal inference in the presence of interference to model interactions among the ads. Quantification of such interactions allows us to better understand the click behavior of users, which in turn impacts the revenue of the host search engine and enhances user satisfaction. We illustrate the utility of our formalization through experiments carried out on the ad placement system of the Bing search engine.

show abstract

Experimental Design in Two-Sided Platforms: An Analysis of Bias

Cited by 32 publications

References 9 publications

Design and Analysis of Bipartite Experiments under a Linear Exposure-Response Model

Design and Analysis of Bipartite Experiments under a Linear Exposure-Response Model

Heterogeneous Treatment Effects for Networks, Panels, and other Outcome Matrices

Causal Inference in the Presence of Interference in Sponsored Search Advertising

Contact Info

Product

Resources

About