What’s the Tone? Easy Doesn’t Do It: Analyzing Performance and Agreement Between Off-the-Shelf Sentiment Analysis Tools

Abstract

This article scrutinizes the method of automated content analysis to measure the tone of news coverage. We compare a range of off-the-shelf sentiment analysis tools to manually coded economic news as well as examine the agreement between these dictionary approaches themselves. We assess the performance of five off-the-shelf sentiment analysis tools and two tailor-made dictionary-based approaches. The analyses result in five conclusions. First, there is little overlap between the off-the-shelf tools; causing wide divergence in terms of tone measurement. Second, there is no stronger overlap with manual coding for short texts (i.e., headlines) than for long texts (i.e., full articles). Third, an approach that combines individual dictionaries achieves a comparably good performance. Fourth, precision may increase to acceptable levels at higher levels of granularity. Fifth, performance of dictionary approaches depends more on the number of relevant keywords in the dictionary than on the number of valenced words as such; a small tailor-made lexicon was not inferior to large established dictionaries. Altogether, we conclude that off-the-shelf sentiment analysis tools are mostly unreliable and unsuitable for research purposesat least in the context of Dutch economic newsand manual validation for the specific language, domain, and genre of the research project at hand is always warranted.

Contested Chinese Dreams of AI? Public discourse about Artificial intelligence on WeChat and People’s Daily Online

“…Because 'off-the-shelf' sentiment dictionaries are commonly used without revalidation, there is no established threshold of correlation for accepting a dictionary. However, by comparing our result with a handful of priors attempts to revalidate off-the-shelf dictionaries (Haselmayer & Jenny, 2016;Boukes et al, 2019), we can argue that our validation result is acceptable. As the correlation is statistically significant, we deem the criterion validity of our measurement to be adequate.…”

Latent Semantic Scaling: A Semisupervised Text Analysis Technique for New Domains and Languages

“…LSS becomes useful when users analyze large corpora, because they otherwise must pay high costs for training with supervised machine learning models or give up pre-defining dimensions of measurement with unsupervised machine learning models. Although LSS predicts scores for individual documents with relatively large errors, their group means are strongly correlated with manual scores, making measurement errors negligible in aggregated-level analysis (Boukes et al, 2020). This became clear when LSS outperformed LSD (r = 0.70 vs. 0.62) and their correlation increased from r = 0.34 to r =0.74 as the size of the samples grew in the longitudinal scaling of economic news.…”

“…However, the possibility of performing dictionary analysis is dependent on the existence of suitable dictionaries in the target domains and languages (Boukes et al, 2020;Grimmer & Stewart, 2013). The English language has the largest collection of dictionaries, such as the Regressive Imagery Dictionary (Martindale, 1975), the Moral Foundation Dictionary (Frimer et al, 2009, the Dictionary of Policy Positions (Laver & Garry, 2000), and Affective Norms for English Words (Nielsen, 2011); in addition to those that I have already mentioned, many of these dictionaries have versions in European languages but not in non-European languages.…”

“…In the following sections, I will summarize the strengths and weaknesses of existing quantitative text analysis methods to highlight the advantages of LSS, explain its concept and algorithm, and demonstrate its flexibility and efficiency using two examples: sentiment analysis of English news articles on the economy and Japanese news articles on politics. Although LSS can be used to scale documents on various dimensions, I choose sentiment because it allows direct comparison between LSS and existing method as it has been a common scale in quantitative text analysis (Boukes et al, 2020;Puschmann & Powell, 2018).…”

Four best practices for measuring news sentiment using ‘off-the-shelf’ dictionaries: a large-scale p-hacking experiment

“…This off-the-shelf dictionary-based sentiment analysis has been used quite heavily in political communication literature (e.g. Boukes et al, 2019;. New dictionaries such as Lexicoder , VADER (Gilbert & Hutto, 2014) and crowd-sourcing-based sentiment dictionaries 1 This paper deals with dictionary-based sentiment analysis only.…”