When Correlation Is Not Enough: Validating Populism Scores from Supervised Machine-Learning Models

Abstract

Despite the ongoing success of populist parties in many parts of the world, we lack comprehensive information about parties’ level of populism over time. A recent contribution to Political Analysis by Di Cocco and Monechi (DCM) suggests that this research gap can be closed by predicting parties’ populism scores from their election manifestos using supervised machine learning. In this paper, we provide a detailed discussion of the suggested approach. Building on recent debates about the validation of machine-learning models, we argue that the validity checks provided in DCM’s paper are insufficient. We conduct a series of additional validity checks and empirically demonstrate that the approach is not suitable for deriving populism scores from texts. We conclude that measuring populism over time and between countries remains an immense challenge for empirical research. More generally, our paper illustrates the importance of more comprehensive validations of supervised machine-learning models.

Synthetically generated text for supervised text analysis

“…Given that no dataset exists that labels populist speech at the sentence level, recent work has proposed training sentence-level supervised classifiers to identify populism using manifesto-level labels (Di Cocco and Monechi 2021). This approach has been criticized, for, among other things, for relying on document-level labels to train a sentence-level classifier when most sentences in a populist party's manifesto will not be recognizably populist (Jankowski and Huber 2023). This application illustrates a new method for identifying populism at the sentence level across 27 European countries in 22 languages.…”

“…and generation hyperparameters used in the prompts, I obtain 36,509 non-populist synthetic sentences. Note that neither the populist nor non-populist text is generated with party names in the prompts, mitigating the risk of the model picking up party names as a predictive feature (Jankowski and Huber 2023). I then train a supervised text classifier on the synthetic sentences.…”

The search for solid ground in text as data: A systematic review and practical recommendations for validation

“…Second, the quantity of validation steps likely corresponds to the quality of the validation procedure, as more validation evidence reduces uncertainty around the validity of the measures, thereby enhancing the trustworthiness of the measures. However, it's important to note that a high count of validation steps doesn't necessarily imply a comprehensive and rigorous validation process (see Jankowski & Huber, 2022). Nevertheless, achieving sufficient validation necessitates the presence of multiple complementary validation steps as a crucial requirement.…”