The Commensurability Problem: Conceptual Difficulties in Estimating the Effect of Behavior on Behavior

Abstract

We pose the commensurability problem: When do the estimates generated by actual research designs correspond to quantities of theoretical interest? We study this question in settings where both treatment and outcome are behavior and the treatment effect of interest is decomposable into direct and informational channels. We establish two results. First, the quantity estimated by an actual research design is only commensurate with the total effect in the ideal experiment if treatment status in the research design is a sufficient statistic for the decision-makers’ information. Second, a research design corresponding to a nonideal experiment isolates just the direct effect in the ideal experiment if two conditions hold: (i) there is no information effect in the nonideal experiment and (ii) the decision-maker’s response function is additively separable in treatment and information. We apply our results to three substantive literatures: the efficacy of protest, the empowerment of female candidates, and indiscriminate violence in counterinsurgency.

A Costly Commitment: Populism, Government Performance, and the Quality of Bureaucracy

“…We also find that populist mayors lead to higher turnover among top bureaucrats, who in turn are on average less educated. We present several robustness tests that reinforce our confidence in the commensurability of our estimates with the effect of electing a populist mayor in close electoral races (Bueno De Mesquita and Tyson 2020).…”

External Validity and Meta‐Analysis

“…While statistical issues can be nested into our framework, our goal is to articulate and stress conceptual or theoretical issues related to meta‐analysis of causal effects. This article thus contributes to an emerging literature on the theoretical implications of empirical models, where scholars have focused on the connection between theory and identification strategies (Abramson, Koçak, and Magazinnik 2019; Bueno de Mesquita and Tyson 2020; Slough 2022; Tomasi 2021), the design of field experiments (Chassang, Padró i Miquel, and Snowberg 2012), and the decision‐theoretic foundations of experiments (Banerjee et al. 2020).…”

“…The set of settings outlines the scope conditions of an argument or theory, that is, when and where we may reasonably expect the mechanism to manifest. Second, a study focuses on a contrast, explicitly or implicitly, which defines the comparison of substantive and empirical interest (Bueno de Mesquita and Tyson 2020). Most commonly, a contrast is given by a treatment/control comparison.…”

Phantom Counterfactuals

“…Ashworth and de Mesquita 2014; Eggers 2017; Gailmard and Patty 2018; Izzo, Dewan, and Wolton 2020; Prato and Wolton 2018). In particular, the identification problems posed by phantom counterfactuals constitute a new class of commensurability problems in which analysts aim to estimate a quantity that is theoretically undefined (Bueno de Mesquita and Tyson 2020). I show how theory can locate these identification problems and suggest remedies in research designs with sequential behavioral outcomes.…”