Cross-Domain Topic Classification for Political Texts

Abstract

We introduce and assess the use of supervised learning in cross-domain topic classification. In this approach, an algorithm learns to classify topics in a labeled source corpus and then extrapolates topics in an unlabeled target corpus from another domain. The ability to use existing training data makes this method significantly more efficient than within-domain supervised learning. It also has three advantages over unsupervised topic models: the method can be more specifically targeted to a research question and the resulting topics are easier to validate and interpret. We demonstrate the method using the case of labeled party platforms (source corpus) and unlabeled parliamentary speeches (target corpus). In addition to the standard within-domain error metrics, we further validate the cross-domain performance by labeling a subset of target-corpus documents. We find that the classifier accurately assigns topics in the parliamentary speeches, although accuracy varies substantially by topic. We also propose tools diagnosing cross-domain classification. To illustrate the usefulness of the method, we present two case studies on how electoral rules and the gender of parliamentarians influence the choice of speech topics.

Natural Language Processing for Policymaking

“…Using the text classification method, we can automate many types of analyses in political science. As listed in the examples in Figure 2, researchers can detect political perspective of news articles (Huguet Cabot et al, 2020), the stance in media on a certain topic (Luo et al, 2020), whether campaigns use positive or negative sentiment (Ansolabehere and Iyengar, 1995), which issue area is the legislation about (Adler and Wilkerson, 2011), topics in parliament speech (Albaugh et al, 2013;Osnabrügge et al, 2021), congressional bills (Hillard et al, 2008;Collingwood and Wilkerson, 2012) and political agenda (Karan et al, 2016), whether the international statement is peaceful or belligerent (Schrodt, 2000), whether a speech contains positive or negative sentiment (Schumacher et al, 2016), and whether a U.S. Circuit Courts case decision is conservative or liberal (Hausladen et al, 2020).…”

Examining the effect of IMF conditionality on natural resource policy

“…Researchers often adopt a weighting scheme, called term frequency-inverse document frequency (TF-IDF), that gives more weight to less frequent words. The main advantage of dictionary methods is the ease of interpretation, while the main disadvantage is the low design efficiency: before conducting any analysis, researchers must spend a significant amount of time designing a classification scheme, by compiling an exhaustive list of keywords that belong to each category (Osnabrügge et al, 2021). A second method for automated text analysis is probabilistic topic modeling.…”

“…A topic is a distribution over a fixed vocabulary (Blei, 2012); for example, the topic natural resources has a fixed vocabulary that includes words like oil , mining , and hydrocarbon . Topic models have high design efficiency (Osnabrügge et al, 2021), because they do not require training sets and are suitable for new discoveries: they can parse the data to identify hidden patterns that are not immediately evident to the human eye (like the unobservable influence of IMF conditionality on domestic legislation).…”

“…The former approach requires researchers to develop a priori expectations about the vocabulary and the topics contained in the documents under study, while the latter can identify hidden patterns that are not immediately evident to the human eye (like the unobservable influence of IMF conditionality on domestic legislation). Dictionary methods have high interpretability and specificity: the resulting classification scheme can be easily interpreted and used to answer specific questions or explore particular data features (Osnabrügge et al, 2021). In contrast, topic models have moderate to low interpretability and specificity, because they require researchers to specify the number of desired topics, label each topic, and interpret the results, all of which are subjective decisions (Wilkerson & Casas, 2017).…”

Media Slant is Contagious

“…We are interested in forming a predicted probability of the source of a document for scoring influence in a second corpus. Other related methods arePeterson and Spirling (2018) andOsnabrügge et al (2021).5 We have fewer snippets from FNC than from CNN/MSNBC. Thus, we randomly under-sample the snippets from the CNN/MSNBC corpus to match the number of snippets from FNC.6 Previous work has shown that supervised learning models using n-grams are rarely sensitive to the specific choices in pre-processing and featurization (e.g.,Denny and Spirling, 2018).…”