Improving massive experiments with threshold blocking

Abstract

Inferences from randomized experiments can be improved by blocking: assigning treatment in fixed proportions within groups of similar units. However, the use of the method is limited by the difficulty in deriving these groups. Current blocking methods are restricted to special cases or run in exponential time; are not sensitive to clustering of data points; and are often heuristic, providing an unsatisfactory solution in many common instances. We present an algorithm that implements a widely applicable class of blocking-threshold blockingthat solves these problems. Given a minimum required group size and a distance metric, we study the blocking problem of minimizing the maximum distance between any two units within the same group. We prove this is a nondeterministic polynomial-time hard problem and derive an approximation algorithm that yields a blocking where the maximum distance is guaranteed to be, at most, four times the optimal value. This algorithm runs in O(n log n) time with O(n) space complexity. This makes it, to our knowledge, the first blocking method with an ensured level of performance that works in massive experiments. Whereas many commonly used algorithms form pairs of units, our algorithm constructs the groups flexibly for any chosen minimum size. This facilitates complex experiments with several treatment arms and clustered data. A simulation study demonstrates the efficiency and efficacy of the algorithm; tens of millions of units can be blocked using a desktop computer in a few minutes.experimental design | blocking | big data | causal inference

We Are All Social Scientists Now: How Big Data, Machine Learning, and Causal Inference Work Together

“…Consider, for example, blocking observations in an experiment—that is, grouping together observations before random assignment to improve the precision of estimated effects. Higgins and Sekhon (2014) leveraged insights from graph theory to provide a blocking algorithm with guarantees about the similarity of observations assigned to the same block. Moore and Moore (2013) used tools to provide a blocking algorithm for experiments that arrive sequentially.…”

Shared Partisanship Dramatically Increases Social Tie Formation in a Twitter Field Experiment

“…From this full list, we constructed a politically balanced set of users to form the subject pool for our experiment; we also removed users with more than 15,000 followers or for whom the partisanship estimator was unable to return a score. To balance subjects across experimental conditions and to improve our precision in our causal inference, we performed randomized assignment by blocking (11). We created homogeneous blocks of users based on (i) users partisanship, (ii) log transform of number of followers (we used log transform since this data is highly skewed), (iii) number of days with at least one tweet in past 14 days (to measure recent activity on the platform), and (iv) number of mutual friendships divided by total number of followers (as a proxy for tendency to reciprocate follows).…”

Rerandomization and Regression Adjustment

“…to conduct completely randomized experiments (CREs) within blocks of covariates. This remains a powerful tool in modern experiments (Miratrix et al, 2013;Higgins et al, 2016;Athey and Imbens, 2017). Blocking is a special case of rerandomization (Morgan and Rubin, 2012), which rejects 'bad' random allocations that violate certain covariate balance criteria.…”