Taking Stock of the Toolkit

Abstract

When analyzing digital journalism content, journalism scholars are confronted with a number of substantial differences compared to traditional journalistic content. The sheer amount of data and the unique features of digital content call for the application of valuable new techniques. Various other scholarly fields are already applying computational methods to study digital journalism data. Often, their research interests are closely related to those of journalism scholars. Despite the advantages that computational methods have over traditional content analysis methods, they are not commonplace in digital journalism studies. To increase awareness of what computational methods have to offer, we take stock of the toolkit and show the ways in which computational methods can aid journalism studies. Distinguishing between dictionary-based approaches, supervised machine learning, and unsupervised machine learning, we present a systematic inventory of recent applications both inside as well as outside journalism studies. We conclude with suggestions for how the application of new techniques can be encouraged.

Content Analysis by the Crowd: Assessing the Usability of Crowdsourcing for Coding Latent Constructs

“…Methodological innovation in content analysis procedures rather focuses on automated, computerassisted techniques (e.g., Boumans & Trilling, 2016;Grimmer & Stewart, 2013;Jacobi, van Atteveldt, & Welbers, 2016). While traditional, manual content analysis is described as a costly and labor intensive methodology (Krippendorff, 2013), the precision and validity with which computers implement content analysis in its various forms is still in development and often depending on the input of human coders in the form of training data for dictionary or algorithm development (e.g.…”

“…Haselmayer & Jenny, 2014) or interpretative coding, that we take into account here. Due to the digital availability of various sorts of texts and increasing computer power to analyze large volumes of texts, great advances have been made in terms of automated, computer-assisted content analysis (e.g., Boumans & Trilling, 2016;Krippendorff, 2013). A common adoption relates to content that clearly manifests itself in the use of pre-defined words and phrases (i.e., dictionaries) (e.g., Pang & Lee, 2008;Riloff & Wiebe, 2003).…”

Political relevance in the eye of the beholder: Determining the substantiveness of TV shows and political debates with Twitter data

“…It should thus be feasible to measure the content of television program on a much larger scale. Automated content analysis of textual data is becoming increasingly common in communication science (e.g., Boumans and Trilling, 2016), but the analysis of television content remains difficult and labor intensive. Applying automated content analysis to "community annotation" data from second-screen users seems to be a promising approach to close this gap.…”

In Validations We Trust? The Impact of Imperfect Human Annotations as a Gold Standard on the Quality of Validation of Automated Content Analysis

“…Although the term "automated content analysis" in general encompasses a wide variety of forms (e.g., Grimmer & Stewart, 2013;Hopkins & King, 2010;Krippendorff, 2013), our definition inevitably excludes automatic approaches of merely acquiring data, data entry, or data management other than the actual coding or classification process (e.g., Lewis, Zamith, & Hermida, 2013). Instead, we concentrate on two broad and rather common formsa dictionary (lexicon-based) approach and a supervised machine learning (SML) approach (see Boumans & Trilling, 2016;Grimmer & Stewart, 2013). As we shall elaborate below, they are rather sensitive to the issue of imperfect gold standards, although the two approaches may nontrivially differ in terms of the degree of their potential sensitivity to this issue.…”