The Sorted Effects Method: Discovering Heterogeneous Effects Beyond Their Averages

Abstract

The partial (ceteris paribus) effects of interest in nonlinear and interactive linear models are heterogeneous as they can vary dramatically with the underlying observed or unobserved covariates. Despite the apparent importance of heterogeneity, a common practice in modern empirical work is to largely ignore it by reporting average partial effects (or, at best, average effects for some groups). While average effects provide very convenient scalar summaries of typical effects, by definition they fail to reflect the entire variety of the heterogeneous effects. In order to discover these effects much more fully, we propose to estimate and report sorted effects—a collection of estimated partial effects sorted in increasing order and indexed by percentiles. By construction, the sorted effect curves completely represent and help visualize the range of the heterogeneous effects in one plot. They are as convenient and easy to report in practice as the conventional average partial effects. They also serve as a basis for classification analysis, where we divide the observational units into most or least affected groups and summarize their characteristics. We provide a quantification of uncertainty (standard errors and confidence bands) for the estimated sorted effects and related classification analysis, and provide confidence sets for the most and least affected groups. The derived statistical results rely on establishing key, new mathematical results on Hadamard differentiability of a multivariate sorting operator and a related classification operator, which are of independent interest. We apply the sorted effects method and classification analysis to demonstrate several striking patterns in the gender wage gap. We find that this gap is particularly strong for married women, ranging from −60normal% to 0normal% between the 2normal% and 98normal% percentiles, as a function of observed and unobserved characteristics; while the gap for never married women ranges from −40normal% to +20normal%. The most adversely affected women tend to be married, do not have college degrees, work in sales, and have high levels of potential experience.

Generic machine learning inference on heterogenous treatment effects in randomized experiments

“…Overall, the heterogeneity patterns that we find are consistent with both glass ceiling theories for the gender wage gap [3], and with heterogenous individual preferences where young, single, and highly educated women are more career-oriented. [20] found similar patterns applying semiparametric regression methods to the same data.…”

“…We first mention work that uses linear and semiparametric regression methods. A semiparametric inference method for characterizing heterogeneity, called the sorted effects method, was given in [20]. This approach does provide a full set of inference tools, including simultaneous bands for percentiles of the CATE, but is strictly limited to the traditional semiparametric estimators for the regression and causal functions.…”

SortedEffects: Sorted Causal Effects in R

“…However, the APE might provide an incomplete summary of ∆ (X) as it neglects all the heterogeneity by design. Chernozhukov et al (2018) proposed the sorted partial effect (SPE) method to provide a more complete summary of ∆ (X). This method consists on reporting the entire set of PEs, sorted in increasing order and indexed by a user-specified ranking in the population of interest.…”

“…To apply the methods in practice, we need to replace ∆ and µ by sample analogs∆ andμ, and quantify the sampling uncertainty. Chernozhukov et al (2018) derived the theoretical underpinnings of the resulting empirical SPEs. We refer the interested reader to that paper for details.…”

Healthy Subjects With Extreme Patterns of Performance Differ in Functional Network Topology and Benefits From Nicotine

“…This would lead to small averaged effects, despite possible systematic, but heterogeneous and non-uniform patterns of effects across nodes. To investigate heterogeneous effects, previous studies suggested to report sorted individual effects in an increasing order and indexed by percentiles (Chernozhukov et al, 2018). Within a first approach on the global level, we aimed to test whether these differences between outliers and nonoutliers, either positive or negative changes, reflect systematic deviations over and above effects of noise.…”