Recalibration of Predicted Probabilities Using the “Logit Shift”: Why Does It Work, and When Can It Be Expected to Work Well?

Abstract

The output of predictive models is routinely recalibrated by reconciling low-level predictions with known quantities defined at higher levels of aggregation. For example, models predicting vote probabilities at the individual level in U.S. elections can be adjusted so that their aggregation matches the observed vote totals in each county, thus producing better-calibrated predictions. In this research note, we provide theoretical grounding for one of the most commonly used recalibration strategies, known colloquially as the “logit shift.” Typically cast as a heuristic adjustment strategy (whereby a constant correction on the logit scale is found, such that aggregated predictions match target totals), we show that the logit shift offers a fast and accurate approximation to a principled, but computationally impractical adjustment strategy: computing the posterior prediction probabilities, conditional on the observed totals. After deriving analytical bounds on the quality of the approximation, we illustrate its accuracy using Monte Carlo simulations. We also discuss scenarios in which the logit shift is less effective at recalibrating predictions: when the target totals are defined only for highly heterogeneous populations, and when the original predictions correctly capture the mean of true individual probabilities, but fail to capture the shape of their distribution.

The Geography of Racially Polarized Voting: Calibrating Surveys at the District Level

“…Existing work proposes to shift each estimate by a constant so that the implied sum matches an election result (Ghitza and Gelman 2013;Rosenman, McCartan, and Olivella 2023). Unfortunately, such a one-way calibration may make the estimates of the racial gap τ white,j − τ black,j even more biased if the estimates of each race-specific vote share are biased in opposite directions.…”

“…This improves weighting estimates to the distribution of race, age, sex, and education within the actual electorate of each district. Furthermore, we develop a two-way survey calibration, which simultaneously calibrates estimates to both election results by geography and an external survey, instead of only to geography (e.g., Ghitza and Gelman 2013;Rosenman, McCartan, and Olivella 2023). This adjustment reduces the bias in the estimates of group voting behavior at the CD level that is due to unobservable selection bias in the survey.…”

Improving Small-Area Estimates of Public Opinion by Calibrating to Known Population Quantities

“…8 The values of these geographic intercepts are chosen such that the model-implied vote shares exactly match the known population vote shares in each geographic unit (Ghitza and Gelman, 2013, 769). This estimator approximates the posterior distribution of cell-level probabilities after conditioning on the ground-truth data (Rosenman, McCartan and Olivella, 2023).…”

“…Methodologically, our paper builds on methods for calibrating model-based inferences to known population quantities. Research in the MRP paradigm commonly uses a correction, some-times called the "logit shift, " to account for discrepancies between known geographic aggregates and model-based estimates (Ghitza and Gelman, 2013;Rosenman, McCartan and Olivella, 2023). This method can improve subgroup estimates for items for which ground-truth data is available, but not at the level of aggregation that is of primary interest.…”

Race and ethnicity data for first, middle, and surnames

“…Sample data processing code are available alongside the count files in the Harvard Dataverse 13 . Code to recreate each of the figures can be found in a separate repository in the Harvard Dataverse 22 .…”