Cross-Lingual Classification of Political Texts Using Multilingual Sentence Embeddings

Abstract

Established approaches to analyze multilingual text corpora require either a duplication of analysts’ efforts or high-quality machine translation (MT). In this paper, I argue that multilingual sentence embedding (MSE) is an attractive alternative approach to language-independent text representation. To support this argument, I evaluate MSE for cross-lingual supervised text classification. Specifically, I assess how reliably MSE-based classifiers detect manifesto sentences’ topics and positions compared to classifiers trained using bag-of-words representations of machine-translated texts, and how this depends on the amount of training data. These analyses show that when training data are relatively scarce (e.g., 20K or less-labeled sentences), MSE-based classifiers can be more reliable and are at least no less reliable than their MT-based counterparts. Furthermore, I examine how reliable MSE-based classifiers label sentences written in languages not in the training data, focusing on the task of discriminating sentences that discuss the issue of immigration from those that do not. This analysis shows that compared to the within-language classification benchmark, such “cross-lingual transfer” tends to result in fewer reliability losses when relying on the MSE instead of the MT approach. This study thus presents an important addition to the cross-lingual text analysis toolkit.

Goodbye Human Annotators? Content Analysis of Policy Debates using ChatGPT

“…More recently, the field has turned to large language models in an effort to improve classification of e.g. ideological placement (Rheault & Cochrane 2020), political emotions (Widmann & Wich 2023) and political manifesto's (Licht 2023;Laurer et al 2024). Miller, Linder & Mebane (2020 explored an "active" labelling strategy where manual classification is aided by a text algorithm to select relevant documents (see also Alshami et al 2023).…”

Measuring and understanding parties' anti-elite strategies

“…A second challenge that arises with cross-country studies is a rapid increase of cases and very often a language barrier (Licht and Lind 2023). Expert surveys and existing content-analytic strategies for measuring parties' anti-elite strategies commonly struggle with one or both of these challenges (cf.…”

“…10 Accordingly, we implemented measures to facilitate the label class balance in our final annotated dataset. First, we distributed only tweets that would likely contain political content for annotation, which we determined by applying a pre-trained supervised text classifier (Licht 2020). 11 Second, we mitigated repeated annotation of posts with very similar content by clustering the remaining tweets into 500 groups based on their multilingual embeddings and sampling tweets from all these "strata".…”

“…To provide benchmarks for the performance of this classifier, we also implemented two alternative approaches: supervised classification using tweet texts' multilingual sentence embeddings (MSEs) as inputs (Licht 2023) and a classic bag-of-words (BoW) classification approach that relies on machine translation (MT) to align tweets' text representations across languages. We discuss the advantages and limitations of these approaches in Section D.1 in the SM.…”

No more cost in translation: Validating open-source machine translation for quantitative text analysis

“…Machine translation (MT) is a popular strategy for researchers who want to apply quantitative text analysis methods to such multilingual text collections (e.g., Baum and Zhukov 2019;Dancygier and Margalit 2020;Barberá et al 2022;cf. Baden et al 2022, Dolinksy et al 2022, Licht and Lind 2023. It allows bridging language barriers by transferring documents written in different languages into a single target language and thus enables researchers to analyze the resulting monolingual documents with standard text-as-data methods (e.g.…”