Modeling and Measuring Deliberation Online

Abstract

Online communication is often characterized as dominated by antagonism or groupthink, with little in the way of meaningful interaction or persuasion. This essay examines how one can detect and measure instances of more productive conversation online, considered through the lens of deliberative theory. It begins with an examination of traditional deliberative democracy, then explores how these concepts have been applied to online deliberation and by those studying interpersonal conversation in social media more generally. These efforts to characterize and measure deliberative quality have resulted in a myriad of criteria, with elaborate checklists that are often as superficial as they are complex. This essay instead proposes targeting what is arguably the core deliberative process—a mutual consideration of conceptually interrelated ideas—in order to distinguish the better from the worse and to construct better conceptual structures. The essay finishes by discussing two computational models of argument quality and interdependence as templates for richer, scalable, nonpartisan measures of deliberative discussion online.

Assessment of Online Deliberative Quality: New Indicators Using Network Analysis and Time-Series Analysis

“…This method is considered a "gold standard," since human experts can extract sophisticated meanings from text [43]. However, Beauchamp [42] has pointed out that it requires intensive work and there may be biases as to what count as deliberative criteria. We agree in part, because there are well-established measures (e.g., Krippendorff's alpha) in content analysis to handle inter-coder reliability.…”

“…By automation, we mean measuring deliberative quality by computational methods rather than by human judgment. According to Beauchamp's review [42], automated methods are still rare but growing, centering around natural language processing and social network analysis. Social network analysis is useful in studying intricate interaction patterns in deliberation processes [44][45][46]65].…”

“…First, the most significant limitation is that human experts can better interpret nuanced political debates and contexts than machines at the current level of technical development. Beauchamp [42] found that studies employing automated methods tend "to focus on superficial, easily measured markers of argument and deliberation" (p. 324), "many of which measures are also disappointingly superficial when examined in detail" (p. 345). Second, he pointed out dozens of heterogenous indicators in the field (p. 336).…”

“…Based on a set of theory-based criteria, most deliberative quality indicators require trained coders to read and assess the quality of the contents of the deliberation [35][36][37][38][39][40][41]. This method is useful for detailed assessment, but it requires significant resources in terms of coders and time [42,43]. In particular, online deliberation often generates thousands and perhaps millions of user-generated discussion data, making human coding schemes difficult.…”

“…We argue that the online deliberative quality could be measured using automated computational methods to provide criteria-based quality information that helps stakeholders and managers of deliberative processes to identify ongoing problems and fix them during the process. More recently, there has been a growing attempt to overcome the limitations of hand-coded measurements by employing automated content analysis and network analysis [42]. These methods are mainly based on digitized text data (e.g., discussion comments) and network data (e.g., discussion threads) [44][45][46][47][48].…”