Some empirical results are more likely to be published than others. Such selective publication leads to biased estimates and distorted inference. This paper proposes two approaches for identifying the conditional probability of publication as a function of a study's results, the first based on systematic replication studies and the second based on meta-studies. For known conditional publication probabilities, we propose median-unbiased estimators and associated confidence sets that correct for selective publication. We apply our methods to recent large-scale replication studies in experimental economics and psychology, and to meta-studies of the effects of minimum wages and de-worming programs.

Much of the debate on the impact of algorithms is concerned with fairness, defined as the absence of discrimination for individuals with the same "merit. " Drawing on the theory of justice, we argue that leading notions of fairness suffer from three key limitations: they legitimize inequalities justified by "merit;" they are narrowly bracketed, considering only differences of treatment within the algorithm; and they consider between-group and not within-group differences. We contrast this fairness-based perspective with two alternate perspectives: the first focuses on inequality and the causal impact of algorithms and the second on the distribution of power. We formalize these perspectives drawing on techniques from causal inference and empirical economics, and characterize when they give divergent evaluations. We present theoretical results and empirical examples which demonstrate this tension. We further use these insights to present a guide for algorithmic auditing and discuss the importance of inequality-and power-centered frameworks in algorithmic decision-making. CCS CONCEPTS• Computing methodologies → Philosophical/theoretical foundations of artificial intelligence; • Social and professional topics → Computing / technology policy.

Standard experimental designs are geared toward point estimation and hypothesis testing, while bandit algorithms are geared toward in‐sample outcomes. Here, we instead consider treatment assignment in an experiment with several waves for choosing the best among a set of possible policies (treatments) at the end of the experiment. We propose a computationally tractable assignment algorithm that we call “exploration sampling,” where assignment probabilities in each wave are an increasing concave function of the posterior probabilities that each treatment is optimal. We prove an asymptotic optimality result for this algorithm and demonstrate improvements in welfare in calibrated simulations over both non‐adaptive designs and bandit algorithms. An application to selecting between six different recruitment strategies for an agricultural extension service in India demonstrates practical feasibility.

Suppose that an experimenter has collected a sample as well as baseline information about the units in the sample. How should she allocate treatments to the units in this sample? We argue that the answer does not involve randomization if we think of experimental design as a statistical decision problem. If, for instance, the experimenter is interested in estimating the average treatment effect and evaluates an estimate in terms of the squared error, then she should minimize the expected mean squared error (MSE) through choice of a treatment assignment. We provide explicit expressions for the expected MSE that lead to easily implementable procedures for experimental design.

We discuss the tension between "what we can get" (identification) and "what we want" (parameters of interest) in models of policy choice (treatment assignment). Our nonstandard empirical object of interest is the ranking of counterfactual policies. Partial identification of treatment effects maps into a partial welfare ranking of treatment assignment policies. We characterize the identified ranking and show how the identifiability of the ranking depends on identifying assumptions, the feasible policy set, and distributional preferences. An application to the project STAR experiment illustrates this dependence. This paper connects the literatures on partial identification, robust statistics, and choice under Knightian uncertainty.

We show how to efficiently use costly testing resources in an epidemic, when testing outcomes can be used to make quarantine decisions. If the costs of false quarantine and false release exceed the cost of testing, the optimal myopic testing policy targets individuals with an intermediate likelihood of being infected. A high cost of false release means that testing is optimal for individuals with a low probability of infection, and a high cost of false quarantine means that testing is optimal for individuals with a high probability of infection. If individuals arrive over time, the policy-maker faces a dynamic trade-off: using tests for individuals for whom testing yields the maximum immediate benefit vs spreading out testing capacity across the population to learn prevalence rates thereby benefiting later individuals. We describe a simple policy that is nearly optimal from a dynamic perspective. We briefly discuss practical aspects of implementing our proposed policy, including imperfect testing technology, appropriate choice of prior, and non-stationarity of the prevalence rate.

Many settings in empirical economics involve estimation of a large number of parameters. In such settings, methods that combine regularized estimation and data-driven choices of regularization parameters are useful. We provide guidance to applied researchers on the choice between regularized estimators and data-driven selection of regularization parameters. We characterize the risk and relative performance of regularized estimators as a function of the data-generating process and show that data-driven choices of regularization parameters yield estimators with risk uniformly close to the risk attained under the optimal (unfeasible) choice of regularization parameters. We illustrate using examples from empirical economics.

No abstract