Matching with Text Data: An Experimental Evaluation of Methods for Matching Documents and of Measuring Match Quality

Abstract

Matching for causal inference is a well-studied problem, but standard methods fail when the units to match are text documents: the high-dimensional and rich nature of the data renders exact matching infeasible, causes propensity scores to produce incomparable matches, and makes assessing match quality difficult. In this paper, we characterize a framework for matching text documents that decomposes existing methods into: (1) the choice of text representation, and (2) the choice of distance metric. We investigate how different choices within this framework affect both the quantity and quality of matches identified through a systematic multifactor evaluation experiment using human subjects. Altogether we evaluate over 100 unique text matching methods along with 5 comparison methods taken from the literature. Our experimental results identify methods that generate matches with higher subjective match quality than current state-of-the-art techniques. We enhance the precision of these results by developing a predictive model to estimate the match quality of pairs of text documents as a function of our various distance scores. This model, which we find successfully mimics human judgment, also allows for approximate and unsupervised evaluation of new procedures in our context. We then employ the identified best method to illustrate the utility of text matching in two applications. First, we engage with a substantive debate in the study of media bias by using text matching to control for topic selection when comparing news articles from thirteen news sources. We then show how conditioning on text data leads to more precise causal inferences in an observational study examining the effects of a medical intervention.

Adjusting for Confounding with Text Matching

“…We see several opportunities for pushing forward the text-based confounding literature. We hope that scholars will extend our work to a proposed alternative to TIRM, a task already started by Mozer et al (2020) and Veitch, Sridhar, and Blei (2019). A central challenge is developing general-purpose methods for evaluating these new models.…”

“…Our approach of augmenting propensity scores with information about topic balance is most similar to the covariate-balancing propensity scores of Imai and Ratkovic (2014). Mozer et al (2020) and Veitch, Sridhar, and Blei (2019) directly build on our framework to propose alternative text adjustment approaches, and the related literature is reviewed in Keith, Jensen, and O'Connor (2020).…”

“…A strength of matching relative to other conditioning strategies is that analysts can evaluate the quality of 1 Our primary contribution is to pose the problem of text-based confounding and offer TIRM as one possible solution. Since our paper started circulating in July 2015, Mozer et al (2020) and Veitch, Sridhar, and Blei (2019) have introduced alternative approaches to text confounding. We hope that there will be further developments in this area.…”

Spherical Regression Under Mismatch Corruption With Application to Automated Knowledge Translation

“…Our method performs substantially better than methods limited to one-to-one correspondence and without using grouping information. Our methodological framework is particularly appealing because it can be extended to a wide range of applications, including confounding adjustment via text matching using text data in social science (Roberts et al 2018, Mozer et al 2018, and cross-language record linkage (Song et al 2016, McNamee et al 2011). The learned mapping matrix Π and translation matrix W have key practical value in transferring statistical models across systems (Torrey & Shavlik 2010), capturing the pose of objects (Zhou et al 2014), estimating the relative angle of proteins (Sael & Kihara 2010) and so on.…”

Identification, Interpretability, and

“…The second will take advantage of stochastic variational inference (Hoffman et al, 2013) to enable Bayesian Word Embeddings to scale to massive corpora. Finally, the third track for future word will involve tying the anchoring approach discussed above with the emerging literature on making casual claims from text (Fong and Grimmer, 2016;Mozer et al, 2018), and taking advantage of the word similarities to identify appropriate linguistic counterfactuals. Figure 4: There is no significant difference between foreign and international topics before and after 1945, uncertainty is displayed with 95% confidence intervals.…”