What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Context

Abstract

Predicting the political bias and the factuality of reporting of entire news outlets are critical elements of media profiling, which is an understudied but an increasingly important research direction. The present level of proliferation of fake, biased, and propagandistic content online, has made it impossible to fact-check every single suspicious claim, either manually or automatically. Alternatively, we can profile entire news outlets and look for those that are likely to publish fake or biased content. This approach makes it possible to detect likely "fake news" the moment they are published, by simply checking the reliability of their source.From a practical perspective, political bias and factuality of reporting have a linguistic aspect but also a social context. Here, we study the impact of both, namely (i) what was written (i.e., what was published by the target medium, and how it describes itself on Twitter) vs. (ii) who read it (i.e., analyzing the readers of the target medium on Facebook, Twitter, and YouTube). We further study (iii) what was written about the target medium on Wikipedia. The evaluation results show that what was written matters most, and that putting all information sources together yields huge improvements over the current state-of-the-art.

We Can Detect Your Bias: Predicting the Political Ideology of News Articles

“…When we use the triplet loss to mitigate the source bias, the resulting article representation is more accurate and meaningful, and the medium representation does offer complementary information, and eventually contributes to sizeable performance gains (see rows 5 and 8 vs. 2). The Twitter bios representation appears to be much more important than the representation from Wikipedia, which shows the importance of inspecting the media followers' background and their point of views, which is also one of the observations in (Baly et al, 2020).…”

Tackling Fake News Detection by Interactively Learning Representations using Graph Neural Networks

“…To evaluate our model's ability to predict the factuality of news medium, we used the Media Bias/Fact Check (MBFC) dataset (Baly et al, 2018(Baly et al, , 2020 (859 sources, each labeled on a 3-point scale based on their factuality: low, mixed, and high). We provide graph statistics in App.…”

“…Table 1 shows our results. We average our models on all 5 data splits released by (Baly et al, 2020), using 20% of the training set sources as a development set, and report results on accuracy and Macro F1-score for fake news source classification. We compare our advice protocol models to the baseline-graph based model trained only on node classification (NC -no advice provided, M4).…”

“…To replicate (Baly et al, 2020) (M3), we used their released code with our features. Specifically, we used our article, Twitter profile, Twitter Follower, and YouTube embeddings.…”

“…Prior works have formulated this as a traditional classification problem using techniques such as feature-based SVM's (Baly et al, 2018(Baly et al, , 2020, and more recently Graph Neural Networks (GNNs) (Li and Goldwasser, 2019;Shu et al, 2019;Han et al, 2020;Nguyen et al, 2020), which create a better representation of social media interactions. Graphs often consist of nodes corresponding to news sources (associated with a discrete factuality level -high, low, or mixed), the articles they release, and their social context, corresponding to social media users engaging and sharing information in their networks.…”