Protest Event Analysis: Developing a Semiautomated NLP Approach

Abstract

Protest event analysis is a key method to study social movements, allowing to systematically analyze protest events over time and space. However, the manual coding of protest events is time-consuming and resource intensive. Recently, advances in automated approaches offer opportunities to code multiple sources and create large data sets that span many countries and years. However, too often the procedures used are not discussed in details and, therefore, researchers have a limited capacity to assess the validity and reliability of the data. In addition, many researchers highlighted biases associated with the study of protest events that are reported in the news. In this study, we ask how social scientists can build on electronic news databases and computational tools to create reliable PEA data that cover a large number of countries over a long period of time. We provide a detailed description our semiautomated approach and we offer an extensive discussion of potential biases associated with the study of protest events identified in international news sources.

Constructing Theory-Informed Relational and Verifiable Protest Event Data

“…Patrick Rafail's ambitious project coding local newspapers from 20 cities has required years of work. More recently, Lorenzini et al (2021) report having 35 graduate students work 642 person-days (5136 hours assuming 8-hour days) for six months reading and coding 45,680 news stories after they were selected from the 5,251,894 that had previously been selected via multiple layers of machine pre-processing. "Easy" articles that describe only one event in a straightforward way can be coded in 5-10 minutes each and screening out articles as irrelevant can usually be done in 2-3 minutes even with a careful read to be sure there are no events lurking in the text.…”

“…Many research teams interested in accurately capturing the characteristics of protests have concluded that the best approaches for now are semi-automated or hybrid workflows that automate the pre-processing of articles to filter out irrelevant articles but rely on human coders at the final steps of identifying and coding events within articles. Major examples include the Mass Mobilization in Autocracies project (Croicu and Weidmann 2015, Hellmeier, Rød and Weidmann 2019, the Cline Center's SPEED project (Nardulli, Althaus and Hayes 2015) and the Zurich-based team interested in political contestation in Europe (Lorenzini et al 2021, Makarov, Lorenzini andKriesi 2016). Our own project has a similar orientation to these hybrid projects and was developed concurrently with them.…”

“…The wellfunded and highly-regarded SPEED project-which trains coders for 70 hours and tests them for accuracy before putting them into production work-reports that tested coders reliably identified 72-85% of all relevant events and accurately coded the information about these events 75-89% of the time (Hayes and Nardulli 2011). The Zurich-based team (Lorenzini et al 2021) reports that their well-trained coders agreed on event identification 60% of the time and recorded Cohen's Kappa scores for coding attributes in the range of .45 to .57 depending on attribute, a range they say is "fair to good." Whether this is good enough partly depends on the research goals, but we can safely conclude that even well-funded projects that invest heavily in coder training have nontrivial rates of inconsistencies in coding, if not outright errors.…”

“…Many teams try to obtain greater agreement about whether an event qualifies for coding with extensive definitions and rules, but there are always ambiguous cases. Lorenzini et al (2021) report dealing with the problem of ambiguity by asking coders to look for a prescribed list of actions (rather than interpreting purposes) but still report only modest inter-coder reliability.…”

“…Difficulties in parsing texts to retrieve events by either humans or machines have often been reported in the methodological literature (e.g., Althaus et al 2017, Boschee, Natarajan and Weischedel 2013, Lorenzini et al 2021. As Nardulli and Hayes (2011:1) say, news articles about events "can be convoluted."…”