Measuring Polarization with Text Analysis: Evidence from the UK House of Commons, 1811–2015

Abstract

Political scientists can rely on a long tradition of applying unsupervised measurement models to estimate ideology and preferences from texts. However, in practice the hope that the dominant source of variation in their data is the quantity of interest is often not realized. In this paper, I argue that in the messy world of speeches we have to rely on supervised approaches that include information on party affiliation in order to produce meaningful estimates of polarization. To substantiate this argument, I introduce a validation framework that may be used to comparatively assess supervised and unsupervised methods, and estimate polarization on the basis of 6.2 million records of parliamentary speeches from the UK House of Commons over the period 1811–2015. Beyond introducing several important adjustments to existing estimation approaches, the paper’s methodological contribution therefore consists of outlining the challenges of applying unsupervised estimation techniques to speech data, and arguing in detail why we should instead rely on supervised methods to measure polarization.

Do the populists have a say? Estimating their effect on topic prevalence and polarization in German state legislature

“…At the beginning of this century, scientists started to use scaling algorithms such as wordfish and wordscore to place party manifestos on an ideological scale (Laver, Benoit, and Garry 2003;Laver and Garry 2000). Today, the research which uses Tzelgov and Olander (2018), and Benoit and Herzog (2017); Schwarz, Traber, and Benoit (2017), Lauderdale and Herzog (2016), and Debus and Bäck (2014); Proksch and Slapin (2012), Proksch and Slapin (2010), and Slapin and Proksch (2008) scaling algorithms Diermeier et al (2012) SVM Gentzkow, Shapiro, and Taddy (2016) custom model polarization Curini, Hino, and Osaka (2018) wordfish Goet (2019), Abercrombie and Batista-Navarro (2018), and Peterson and Spirling (2018) text classifier Spirling, Huang, and Patrick (2018) bayesian Rheault and Cochrane (2019) word embeddings sentiment Rheault, Beelen, et al (2016) GloVe Proksch, Lowe, et al (2018) Multilingual dictionary floor time Blumenau (2019) and Bäck, Debus, and Müller (2014) regression topical prevalence Høyland and Søyland (2019) and Greene and Cross (2017) topic models…”

“…However, with advances in the fields of computer science, i.e. natural language processing, and linguistics, political scientists have started to explore classification algorithms (Peterson and Spirling 2018;Goet 2019; Abercrombie and Batista-Navarro 2018), topic models (Høyland and Søyland 2019;Greene and Cross 2017) and word embeddings for their research (Rheault and Cochrane 2019;Rheault, Beelen, et al 2016).…”

Shirking and Slacking in Parliament

“…To do so, we draw on all speech records from the lower and upper houses at the federal level for the period 1999–2015. We estimate how close legislators are to their party using a novel machine‐learning approach proposed by Peterson and Spirling (2018) (see also Goet 2019). This approach fits a machine classifier to a training sample of speeches from a party and uses the trained model to predict a held‐out sample of speeches.…”

Comunistas? Uma análise dos projetos de lei dos deputados federais do PCdoB e PPS

“…Embora não seja o objetivo principal da investigação, a classificação ideológica dos partidos, há a indicação, em diferentes trabalhos, do modus operandi para essa análise a partir de seus dados. Com isso, muitos investigadores têm utilizado essa metodologia para realizar classificações em termos ideológicos dos partidos políticos (LORENZONI;PÉREZ, 2013;GOET, 2019;ECKER;MEIER, 2019;ÖZTÜRK GÖKTUNA, 2019). A metodologia do MP segue uma análise quantitativa de conteúdo dos programas eleitorais de agremiações e está dividida em duas fases.…”