Creating and Comparing Dictionary, Word Embedding, and Transformer-Based Models to Measure Discrete Emotions in German Political Text

Abstract

Previous research on emotional language relied heavily on off-the-shelf sentiment dictionaries that focus on negative and positive tone. These dictionaries are often tailored to nonpolitical domains and use bag-of-words approaches which come with a series of disadvantages. This paper creates, validates, and compares the performance of (1) a novel emotional dictionary specifically for political text, (2) locally trained word embedding models combined with simple neural network classifiers, and (3) transformer-based models which overcome limitations of the dictionary approach. All tools can measure emotional appeals associated with eight discrete emotions. The different approaches are validated on different sets of crowd-coded sentences. Encouragingly, the results highlight the strengths of novel transformer-based models, which come with easily available pretrained language models. Furthermore, all customized approaches outperform widely used off-the-shelf dictionaries in measuring emotional language in German political discourse.

A Systematic Evaluation of Text Mining Methods for Short Texts: Mapping Individuals’ Internal States from Online Posts

“…We find that advanced supervised machine learning classification methods using transformer language models can approach the performance of human analysis when it comes to inference on various internal states from short texts. Our results, thus, echo recent suggestions about the potential of deep learning methods in social science applications (van Atteveldt et al 2021;Bonikowski, Luo, and Stuhler 2022;Do, Ollion, and Shen 2022;Widmann and Wich 2022). Yet, we also suggest that increased method complexity does not always warrant a large improvement in performancesimple supervised machine learning methods such as logistic regression can sometimes perform almost as well as more complex algorithms.…”

“…An epoch denotes an iteration during which the model has used all the relevant data for learning coding patterns once; when completed, the model is updated based on the data it has seen and then the data can be passed through the updated model again, training it for another epoch and letting it learn from the data even further. We then choose the bestperforming epoch for each model (Widmann and Wich 2022).…”

“…Guided by this framework, we present a comprehensive benchmark of approaches from three families of automatic text analysis methods: dictionary methods, supervised machine learning classification methods, and unsupervised machine learning clustering methods. Building on existing work (Barberá et al 2021;Nelson et al 2021;Widmann and Wich 2022), we survey the performance of various algorithms within these three families across four different datasets and coding tasks.…”

Introducing an Interpretable Deep Learning Approach to Domain-Specific Dictionary Creation: A Use Case for Conflict Prediction

“…Second, we assess its performance against two widely used document scaling techniques: Wordscores (Laver, Benoit, and Garry 2003) and Wordfish (Lo, Proksch, and Slapin 2016). Finally, as transformer architectures are considered the current state-of-the-art technique in NLP (see, e.g., Widmann and Wich 2022), the performance of our dictionary is compared to the performance of the newly released ConfliBERT model (Hu et al 2022). As a measure of performance, we investigate alignment with conflict trends over time and correlations with our variable of interest.…”

“…They are able to show considerable improvements of their approach compared with other dictionaries. Similarly, Widmann and Wich (2022) apply a word embedding model and manual coding to extend an existing German-language sentiment dictionary. They compare this dictionary to word embeddings and transformer models, finding that transformer models outperform the other approaches.…”