Adjusting for Confounding with Text Matching

Abstract

We identify situations in which conditioning on text can address confounding in observational studies. We argue that a matching approach is particularly well-suited to this task, but existing matching methods are ill-equipped to handle high-dimensional text data. Our proposed solution is to estimate a low-dimensional summary of the text and condition on this summary via matching. We propose a method of text matching, topical inverse regression matching, that allows the analyst to match both on the topical content of confounding documents and the probability that each of these documents is treated. We validate our approach and illustrate the importance of conditioning on text to address confounding with two applications: the effect of perceptions of author gender on citation counts in the international relations literature and the effects of censorship on Chinese social media users. Verification Materials: The materials required to verify the computational reproducibility of the results, procedures, and analyses in this article are available on the American Journal of Political Science Dataverse within the Harvard Dataverse Network, at: https://doi.org/10.7910/DVN/HTMX3K. S ocial media users in China are censored every day, but it is largely unknown how the experience of being censored affects their future online experience. Are social media users who are censored for the first time flagged by censors for increased scrutiny in the future? Is censorship "targeted" and "customized" toward specific users? Do social media users avoid writing after being censored? Do they continue to write on sensitive topics or do they avoid them? Experimentally manipulating censorship would allow us to make credible causal inferences about the effects of experiencing censorship, but this is impractical

Matching with Text Data: An Experimental Evaluation of Methods for Matching Documents and of Measuring Match Quality

“…STM-based representations appear somewhat more sensitive to tuning parameters, with representations that include a large number of estimated topics achieving higher average match quality than those constructed with fewer topics. This result provides further support for the findings in Roberts, Stewart, and Nielsen (2019), where the authors found that matching on more topics generally led to better results in terms of recovering pairs of nearly identical documents.…”

“…Text matching is widely applicable in the social sciences. Roberts, Stewart, and Nielsen (2019) show how text matching can produce causal estimates in applications such as international religious conflict, government-backed internet censorship, and gender bias in academic publishing. We believe that the framework presented in this paper will help expand the scope and usability of text matching even further and will facilitate investigation of text data across a wide variety of disciplines.…”

“…These matched documents can then be used to make unbiased comparisons between groups on external features, such as rates of citation or online views, or on features of the text itself, such as sentiment. In the case where group membership can be thought of as the receipt of a particular intervention (e.g., documents that were censored vs. not, such as in Roberts, Stewart, and Nielsen (2019)), this allows us to draw causal inferences about effects of interest.…”

“…Our work builds on Roberts, Stewart, and Nielsen (2019), the seminal paper in this literature, which introduces text matching and describes a procedure for matching documents on a lower-dimensional representation of text based on a structural topic model (STM) (Roberts, Stewart, and Airoldi 2016). They also present several applications that motivate the use of text matching to address confounding and describe several of the methodological challenges for matching that arise in these settings.…”

“…Building upon this work, we develop a general framework for constructing and evaluating text-matching methods. This allows us to consider a number of alternative matching methods not considered in Roberts, Stewart, and Nielsen (2019), each characterized by one representation of the corpus and one distance metric. Within this framework, we also present a systematic approach for comparing different matching methods through a human evaluation experiment, which identifies methods that produce more matches and/or matches of higher quality than those produced by TIRM.…”