Stay on Topic, Please: Aligning User Comments to the Content of a News Article

Abstract

Social scientists have shown that up to 50% of the comments posted to a news article have no relation to its journalistic content. In this study we propose a classification algorithm to categorize user comments posted to a news article based on their alignment to its content. The alignment seeks to match user comments to an article based on similarity of content, entities in discussion, and topics. We propose a BERTAC, BERT-based approach that learns jointly article-comment embeddings and infers the relevance class of comments. We introduce an ordinal classification loss that penalizes the difference between the predicted and true labels. We conduct a thorough study to show influence of the proposed loss on the learning process. The results on five representative news outlets show that our approach can learn the comment class with up to 36% average accuracy improvement comparing to the baselines, and up to 25% comparing to the BA-BC. BA-BC is our approach that consists of two models aimed to capture dis-jointly the formal language of news articles and the informal language of comments. We also conduct a user study to evaluate human labeling performance to understand the difficulty of the classification task. The user agreement on comment-article alignment is "moderate" per Krippendorff's alpha score, which suggests that the classification task is difficult. Keywords: Text mining • Text classification • Online news • News comments • Relevancy • Understanding user-generated text J. Alshehri and M. Stanojevic-contributed equally.

Online Political Comments: Americans Talk About the Election Through a “Horse-Race” Lens

“…Overwhelmingly, we found that most comments were neither about horse-race or policy, supporting research that shows that a majority of comments on news stories are unrelated to the journalistic content [15]. This is troubling but unsurprising because it suggests people are not using news sites to comment or discuss the election in meaningful ways.…”

“…Election coverage frequently falls neatly into either horse-race or policy categories [3] [4], but our data showed that online discussions clearly do not. This is unsurprising but informative because online commenters often veer off-topic [15]. Examples of comments that fit into "neither" included nonsensical content (e.g., "whaaaaaa.....whaaaaaaaaaa"), responses to other commenters that had little content (e.g., "LOL"), impolite speech (e.g., "F OFF LOSER"), and off-topic content (e.g., "Did Katy Perry jump off a NY skyscraper this morning?…”

“…Although some research has found relationships between the content of news stories and the content of comments posted on the stories [29], there is no way to ascertain whether commenters or those reading comments actually read the news story. Indeed, evidence suggests that many people comment without reading the stories [9], and research has shown that as many as half of comments are unrelated to the content of the stories upon which they are posted [15]. The problem of people commenting without reading a story is so prominent that one news organization is requiring that users pass a quiz about the story before they can comment [30] to help ensure commenters are discussing the story.…”

“…This study offers several notable contributions to our understanding of online political communication. From a normatively negative standpoint, we found that most comments had nothing to do with horse-race or policy elements of the election but were in fact were offtopic, vitriolic, or non-substantive content commonly found online [14] [15]. Yet, these conversations focused more frequently on horse-race elements than policies, which might increase their distrust, cynicism, or polarization [5] [6] [7].…”

Aligning Comments to News Articles on a Budget

“…In many practical problems, humans do not agree on their annotations [4], providing noisy labels that affect the downstream performance of machine learning models [8]. One such problem is the Article-Comment Alignment Problem (ACAP) [2], where the goal is to classify an article-comment pair into one of several categorical relevancy levels. For example, an annotator might consider article-comment pairs relevant, while another might consider them irrelevant.…”