NELA-GT-2019: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles

Gruppi, Maurício; Horne, Benjamin D.; Adalı, Sibel

doi:10.48550/arxiv.2003.08444

Cited by 6 publications

(9 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In terms of content, the data collections most similar to the MIND corpus are the Emergent [12], NELA-GT Series [13,14,15] and FacebookHoax [9] datasets. The Emergent collection comprises news articles covering war conflicts, politics and business/technology, and it was created to study how online media handled with unverified information.…”

Section: Related Workmentioning

confidence: 99%

MINT -- Mainstream and Independent News Text Corpus

Caled¹,

Carvalho²,

Silva³

2021

Preprint

View full text Add to dashboard Cite

This paper presents and characterizes MIND, a new Portuguese corpus comprised of different types of articles collected from online mainstream and alternative media sources, over a full year period. The articles in the corpus are organized into five collections: facts, opinions, entertainment, satires, and conspiracy theories. Throughout this paper, we explain how the data collection process was conducted, and present a set of linguistic metrics that allow us to perform a preliminary characterization of the texts included in the corpus. Also, we deliver an analysis of the most frequent topics in the corpus, and discuss the main differences and similarities among the collections considered. Finally, we enumerate some tasks and applications that could benefit from this corpus, in particular the ones (in)directly related to misinformation detection. Overall, our contribution of a corpus and initial analysis are designed to support future exploratory news studies, and provide a better insight into misinformation.

show abstract

Section: Related Workmentioning

confidence: 99%

MINT -- Mainstream and Independent News Text Corpus

Caled¹,

Carvalho²,

Silva³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Wang (2017); Shu et al (2017) collected manually labeled statements or news articles from fact-checking websites. The NELA datasets (Horne et al, 2018b;Nørregaard et al, 2019;Gruppi et al, 2020) scrape news articles directly from news outlets and use the manually annotated labels from Media Bias/Fact Check (MBFC) as site-level annotations. Social media is also a popular resource for collecting news stories (Nakamura et al, 2020;Santia and Williams, 2018;Mitra and Gilbert, 2015).…”

Section: Related Workmentioning

confidence: 99%

“…Thus we focus our analysis on other potentially confounding factors in the dataset. We use the latest aggregated site-level labels provided in NELA-GT-2019 (Gruppi et al, 2020) and report both the article-and site-level accuracy. For article-level accuracy, we assign the site-level label to all articles from that news outlet and calculate per-article accuracy.…”

Section: Logistic Regression (Lr)mentioning

confidence: 99%

“…To test the effect of time, we examine the model's performance on news articles from a temporally disjoint dataset. Specifically, since all our models are trained on the NELA-GT-2018 (Nørregaard et al, 2019), we use the NELA-GT-2019 (Gruppi et al, 2020) as the evaluation dataset. We split the news articles in 2019 into twelve months and plot the performance trend in Figure 1.…”

Section: Effect Of Split By Timementioning

confidence: 99%

See 1 more Smart Citation

Hidden Biases in Unreliable News Detection Datasets

Zhou¹,

Elfardy²,

Christodoulopoulos³

et al. 2021

Preprint

View full text Add to dashboard Cite

Automatic unreliable news detection is a research problem with great potential impact. Recently, several papers have shown promising results on large-scale news datasets with models that only use the article itself without resorting to any fact-checking mechanism or retrieving any supporting evidence. In this work, we take a closer look at these datasets. While they all provide valuable resources for future research, we observe a number of problems that may lead to results that do not generalize in more realistic settings. Specifically, we show that selection bias during data collection leads to undesired artifacts in the datasets. In addition, while most systems train and predict at the level of individual articles, overlapping article sources in the training and evaluation data can provide a strong confounding factor that models can exploit. In the presence of this confounding factor, the models can achieve good performance by directly memorizing the site-label mapping instead of modeling the real task of unreliable news detection. We observed a significant drop (>10%) in accuracy for all models tested in a clean split with no train/test source overlap. Using the observations and experimental results, we provide practical suggestions on how to create more reliable datasets for the unreliable news detection task. We suggest future dataset creation include a simple model as a difficulty/bias probe and future model development use a clean non-overlapping site and date split. 1

show abstract

“…Most existing datasets for predicting the political ideology at the news article level were created by crawling the RSS feeds of news websites with known political bias (Kulkarni et al, 2018), and then projecting the bias label from a website to all articles crawled from it, which is a form of distant supervision. The crawling could be also done using text search APIs rather than RSS feeds (Horne et al, 2019;Gruppi et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

We Can Detect Your Bias: Predicting the Political Ideology of News Articles

Baly¹,

Martino²,

Glass³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

We explore the task of predicting the leading political ideology or bias of news articles. First, we collect and release a large dataset of 34,737 articles that were manually annotated for political ideology -left, center, or right-, which is well-balanced across both topics and media. We further use a challenging experimental setup where the test examples come from media that were not seen during training, which prevents the model from learning to detect the source of the target news article instead of predicting its political ideology. From a modeling perspective, we propose an adversarial media adaptation, as well as a specially adapted triplet loss. We further add background information about the source, and we show that it is quite helpful for improving article-level prediction. Our experimental results show very sizable improvements over using state-of-the-art pre-trained Transformers in this challenging setup.

show abstract

NELA-GT-2019: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles

Cited by 6 publications

References 1 publication

MINT -- Mainstream and Independent News Text Corpus

MINT -- Mainstream and Independent News Text Corpus

Hidden Biases in Unreliable News Detection Datasets

We Can Detect Your Bias: Predicting the Political Ideology of News Articles

Contact Info

Product

Resources

About