Enhancing Theory-Informed Dictionary Approaches with “Glass-box” Machine Learning: The Case of Integrative Complexity in Social Media Comments

Dobbrick, Timo; Jakob, Julia; Chan, Chung-hong; Weßler, Hartmut

doi:10.1080/19312458.2021.1999913

Cited by 8 publications

(7 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our study thus complements existing research about how to automatise the detection of important dimensions of debate quality (e.g. Dobbrick et al, 2022; Fournier-Tombs and MacKenzie, 2021).…”

Section: Introductionsupporting

confidence: 57%

“…Different supervised machine learning classifiers trained on different sets of (open or closed vocabulary) features were evaluated for out-of-sample label prediction and generalisability to new contexts. Dobbrick et al (2022) further proposed to combine off-the-shelf dictionaries with supervised machine learning to locate the major source of weakness in the dictionary approach and, thereby, to improve the detection of integrative complexity. Furthermore, Fournier-Tombs and MacKenzie (2021) have proposed to rely on machine learning to study discourse quality covering dimensions of an adapted version of the DQI.…”

Section: Study Backgroundmentioning

confidence: 99%

“…However, in today’s communication environment, both parliamentary and online debate are indispensable channels of communication for spreading political messages. Against this background, it is important to gain knowledge about what are reliable measures of quality in political debate on social media (Dobbrick et al, 2022). To date, changes in debate quality have essentially been investigated within offline political communication arenas, albeit there are a few exceptions (e.g.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Comparing and mapping difference indices of debate quality on Twitter

Reveilhac¹

2023

Methodological Innovations

View full text Add to dashboard Cite

Albeit the measurement of debate quality is not a new endeavour, this paper raises two research questions for which we still have limited knowledge: What are important and reliable indicators of debate quality on social media? How does debate quality relate to individual factors on social media? First, we empirically analysed how two well-established discourse’ quality indices (the DQI and CC index) correlate to each other using a random sample of 1000 tweets selected from the full history of tweets written by Swiss elected politicians between 2011 and 2021. While the sample was automatically coded for CC using LIWC, we manually annotated the tweets according to an adapted version of the DQI for social media texts. Second, we conducted a correspondence analysis to investigate the relations between these dimensions, additional debate quality features, as well as individual political factors. Results show a good correlation between both indices ( r up to 0.46), while also highlighting their respective weaknesses. Furthermore, the results highlight the necessity to include alternative dimensions of debate quality (such as emotion and inclusive or exclusive views) to enhance future measurements of debate quality in the realm of social media.

show abstract

Section: Introductionsupporting

confidence: 57%

Section: Study Backgroundmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Comparing and mapping difference indices of debate quality on Twitter

Reveilhac¹

2023

Methodological Innovations

View full text Add to dashboard Cite

show abstract

“…We combined an off-the-shelf dictionary with machine learning to measure toxic outrage in the collected posts—a novel automated method suggested by Dobbrick et al (2021). This leveraged the knowledge incorporated in the word list and tailored it to our needs, thus reducing the amount of hand-coded data required for stand-alone machine learning.…”

Section: Methodsmentioning

confidence: 99%

Is Constructive Engagement Online a Lost Cause? Toxic Outrage in Online User Comments Across Democratic Political Systems and Discussion Arenas

Jakob

Dobbrick

Freudenthaler

et al. 2022

Communication Research

Self Cite

View full text Add to dashboard Cite

This study is the first to simultaneously investigate country-level and platform-related context factors of toxic outrage, that is, destructive incivility, in online discussions. It compares user comments on the public role of religion and secularism from 2015/16 in four democracies (Australia, United States, Germany, Switzerland) and four discussion arenas on three platforms (News websites, Facebook, Twitter). A novel automated content analysis ( N = 1,236,551) combines LIWC dictionaries with machine learning. The level of toxic outrage is higher in majoritarian than in consensus-oriented democracies and in arenas that afford plural, issue-driven rather than like-minded, preference-driven debates. Yet, toxic outrage is lower in forums that tend to separate public and private conversations than in those that collapse varying contexts. This suggests that user-generated discussions flourish in environments that incentivize actors to strive for compromise, put relevant issues center stage and make room for public debate at a relative distance from purely social conversation.

show abstract

“…TO ERR IS CONTENT ANALYSIS; TO AUTOMATE, DILUTE 4 procedures (Dobbrick et al, 2021;González-Bailón & Paltoglou, 2015). Van Atteveldt and Peng (2018) specify that "even if a researcher uses an existing off-the-shelf tool with published validity results it is vital to show how well it performs in a specific domain and on a specific task."…”

mentioning

confidence: 99%

Automation-coerced, increased dilution of correlation

Chan¹

2022

Preprint

View full text Add to dashboard Cite

Automated data-making methods in content analysis —like all measurements— are fallible. The purpose of this simulation study is to show this fallibility can lead to the correlation dilution effect: the biased estimation of true effect size towards zero, or, in other words, the unexpected reduction in statistical power. An alternative way to measure the performance of automated procedures, which focuses on the retention of statistical power, is proposed. This paper ends with best practices regarding planning, executing, and reporting of automated content analyses.

show abstract

Enhancing Theory-Informed Dictionary Approaches with “Glass-box” Machine Learning: The Case of Integrative Complexity in Social Media Comments

Cited by 8 publications

References 32 publications

Comparing and mapping difference indices of debate quality on Twitter

Comparing and mapping difference indices of debate quality on Twitter

Is Constructive Engagement Online a Lost Cause? Toxic Outrage in Online User Comments Across Democratic Political Systems and Discussion Arenas

Automation-coerced, increased dilution of correlation

Contact Info

Product

Resources

About