Algorithm is Experiment: Machine Learning, Market Design, and Policy Eligibility Rules

Abstract

Algorithms produce a growing portion of decisions and recommendations both in policy and business. Such algorithmic decisions are natural experiments (conditionally quasirandomly assigned instruments) since the algorithms make decisions based only on observable input variables. We use this observation to develop a treatment-effect estimator for a class of stochastic and deterministic algorithms. Our estimator is shown to be consistent and asymptotically normal for well-defined causal effects. A key special case of our estimator is a high-dimensional regression discontinuity design. The proofs use tools from differential geometry and geometric measure theory, which may be of independent interest.The practical performance of our method is first demonstrated in a high-dimensional simulation resembling decision-making by machine learning algorithms. Our estimator has smaller mean squared errors compared to alternative estimators. We finally apply our estimator to evaluate the effect of Coronavirus Aid, Relief, and Economic Security (CARES) Act, where more than $10 billion worth of relief funding is allocated to hospitals via an algorithmic rule. The estimates suggest that the relief funding has little effects on COVID-19-related hospital activity levels. Naive OLS and IV estimates exhibit substantial selection bias.

Counterfactual Learning with General Data-Generating Policies

“…1 Method. Our OPE estimator is based on a modification of the Propensity Score (Rosenbaum and Rubin 1983), which we dub the "Approximate Propensity Score" (APS) (Narita and Yata 2022). APS of action (arm) a at context (covariate) value x is the average probability that the logging policy chooses action a over a shrinking neighborhood around x in the context space.…”

“…Our approach instead predicts the mean reward differences between actions by exploiting local subsamples near the decision boundaries without specifying the regression model. Narita and Yata (2022) originally develop and empirically apply this approach in the context of treatment effect estimation with a binary treatment. This paper extends their approach to OPE with multiple actions.…”