Heterogeneous Causal Effects and Sample Selection Bias

Abstract

Abstract:The role of education in the process of socioeconomic attainment is a topic of long standing interest to sociologists and economists. Recently there has been growing interest not only in estimating the average causal effect of education on outcomes such as earnings, but also in estimating how causal effects might vary over individuals or groups. In this paper we point out one of the under-appreciated hazards of seeking to estimate heterogeneous causal effects: conventional selection bias (that is, selection on baseline differences) can easily be mistaken for heterogeneity of causal effects. This might lead us to find heterogeneous effects when the true effect is homogenous, or to wrongly estimate not only the magnitude but also the sign of heterogeneous effects. We apply a test for the robustness of heterogeneous causal effects in the face of varying degrees and patterns of selection bias, and we illustrate our arguments and our method using National Longitudinal Survey of Youth 1979 (NLSY79) data. data.Keywords: causality; sample selection bias; heterogeneous treatment effects; propensity score; education; NLSY79 T HE role of education in the process of socioeconomic attainment is a topic of longstanding interest to sociologists and economists. Following the causal revolution of the past thirty years in the social sciences, there is a growing interest not only in estimating the causal effect of education on outcomes, such as earnings, but also in understanding and estimating how these effects differ among individuals (Brand and Xie 2010 provide a sociological example, Carneiro, Heckman, and Vytlacil 2011 an economic one). In this paper, drawing on earlier work by Heckman and Navarro-Lozano (2004), we point out one of the under-appreciated hazards of seeking to estimate heterogeneous causal effects: conventional selection bias (that is, selection on baseline differences) can easily be confused with heterogeneity of causal effects. One consequence is that analyses might find heterogeneous effects when the true effect is homogenous; another is that we might wrongly estimate either the direction or magnitude, or both, of a truly heterogeneous effect.We develop this argument initially using the potential outcomes approach to explain what we mean by selection on baseline differences and by heterogeneous causal effects. Our focus here is on heterogeneity of the causal effects according to the probability of receiving treatment. Then we show how, in both an important special case and in general, selection may confound estimates of heterogeneous causal effects, or, more simply, how selection on baseline differences may easily be mistaken for heterogeneous causal effects. Next we describe a test that allows us to calculate the degree to which estimates of heterogeneous causal effects are robust to different types and kinds of baseline selection bias. We apply this method

Parental divorce is not uniformly disruptive to children’s educational attainment

“…If ignorability is true, we may interpret the pattern in the effect of divorce as a function of the likelihood, or propensity, of disruption. However, if ignorability does not hold, such that we have heterogeneous responses to latent determinants of divorce, the same results are still interpretable because they indicate variation in effects of parental divorce by the latent unobserved parental resistance to divorce, a consideration missed in a critique of this approach (21). That is, we assume that lower observed propensity for divorce is associated with lower unobserved resistance to divorce, with lower resistance meaning that parents choose divorce despite potential negative effects for children's well-being.…”

“…One key advantage and primary motivation for our focus on treatment effect heterogeneity by the propensity score is the heightened recognition of potential violations of the assumption that we adequately adjust for all potential confounding factors. A researcher can begin with such an assumption to carry out meaningful analyses without necessarily committing to the validity of the assumption (19)(20)(21). Indeed, even when unobserved selectivity is present, it is informative to understand variation in effects along the propensity score (22).…”

Unequal Families, Unequal Effects: How Parental Divorce Differentially Impacts Children's Educational Attainment

“…Reversing the findings from Brand and Xie (2010), for example, depends upon a result in which there is an assumed negative effect of college on wages for low propensity college goers, a substantive result with little if any support from the literature in economics or sociology. Although Breen, Choi, and Holm (2015) take up no substantive issues that provide insight into these processes, Brand and Xie (2010) emphasize both selectivity bias and substantive interpretations, as we do here. We cannot, nor can Breen, Choi, and Holm (2015), adjudicate between selection and substantive interpretations.…”

“…Although Breen, Choi, and Holm (2015) take up no substantive issues that provide insight into these processes, Brand and Xie (2010) emphasize both selectivity bias and substantive interpretations, as we do here. We cannot, nor can Breen, Choi, and Holm (2015), adjudicate between selection and substantive interpretations. See Zhou and Xie (2016aXie ( , 2016b for additional discussion on this issue.…”

Parents Who Left College and Children's Postsecondary Educational Attainment

“…In the main, this work shows that adults who have some postsecondary education experience a gain in these outcomes compared to peers with only a high school diploma or general education degree, but a premium accrues to those who hold a bachelor's degree (Isaacs et al 2012;Julian and Kominski 2011). Further, there is some evidence that the economic gains to degree attainment are strongest among those who are least likely to attend college (Brand and Xie 2010), although support for this relationship has been tempered by the argument that apparent heterogeneous treatment effects may be attributable to variation in selection bias over levels of the propensity to complete college (Breen, Choi, and Holm 2015). Thus, the family economic circumstances of children whose parents have some college may represent a real gain over children whose parents have completed only high school or a General Education Development certificate (GED), but this distinction is marginal in comparison to the disparity in lifetime earnings between those with a four-year college degree and those without.…”