What is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory

Abstract

The link between theory and quantitative empirical evidence is a longstanding hurdle in sociological research. Ambiguity about the role that statistical evidence plays in an argument may produce misleading conclusions and poor methodological practice. This ambiguity could be reduced if researchers would state the theoretical estimand---the central quantity at the core of a given paper---in precise language. Our approach envisions three choices in the research process: (1) choice of a theoretical estimand, which will be informative for theory, (2) choice of an empirical estimand, which is informative about the theoretical estimand under some identification assumptions, and (3) choice of an estimation strategy to learn the empirical estimand from data. Key advantages of this approach include improved clarity on the object of interest, transparency about how empirical evidence contributes to knowledge of that quantity, and the ability to easily plug in new statistical tools for estimation.

Comment: Summarizing Income Mobility with Multiple Smooth Quantiles Instead of Parameterized Means

“…The two choices we raise-distributional summary and functional form-are applicable in any regression context. Researchers often equate the research goal with the coefficient of a regression model, but we advocate a more conscious choice of estimand (Lundberg, Johnson, and Stewart 2020). Research constrained to the study of parameterized means risks obscuring important sources of evidence.…”

Machine Learning for Social Science: An Agnostic Approach

“…The focus onβ arises from an interest in mapping the (often causal) relationships between variables. Broadly speaking, machine learning can be helpful in this area as well-particularly if we abandon the idea that the relationship between two variables needs to be represented by a single parameter in a linear model (Lundberg et al 2021)-but it requires refocusing on the specific tasks relevant to social science.…”

The Only Thing That Can Stop Bad Causal Inference Is Good Causal Inference

“…Without any actual group differences in aggressiveness, this means that --conditional on police exposure --Black individuals involved in such situations are less aggressive, which would decrease their chances of being fatally shot. Such induced confounding could hide decision-maker bias and has been discussed at great length (see Hu, 2021;and Lundberg et al, 2020 for summaries of the debate); it crops up for other topics as well (e.g., the gender wage gap, Hünermund, 2018). Outside of the lab, individuals are not randomly allocated to situations, which makes it challenging to identify decision-maker bias in observational data.…”