NELA-GT-2022: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles

Gruppi, Maurício; Horne, Benjamin D.; Adalı, Sibel

doi:10.48550/arxiv.2203.05659

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The reiteration dataset is created using headlines from the MisinfoReactionFrames dataset . We use false headlines that fall into two topics: climate change, which come from NELA-GT-2018-2020 (Gruppi et al, 2022;Nørregaard et al, 2019) and ClimateFEVER (Bulian et al, 2020); as well as COVID, which come from The CoronaVirusFacts/DatosCoronaVirus Alliance Database 112 , and the ESOC Covid19 Misinformation Dataset 113 . 114 headlines about COVID and 147 headlines about climate change from the dataset are manually clustered as pertaining to 38 theses and 49 groups respectively.…”

Section: E5 Disinformationmentioning

confidence: 99%

Holistic Evaluation of Language Models

Liang¹,

Bommasani²,

Lee³

et al. 2022

Preprint

View full text Add to dashboard Cite

Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models. First, we taxonomize the vast space of potential scenarios (i.e. use cases) and metrics (i.e. desiderata) that are of interest for LMs. Then we select a broad subset based on coverage and feasibility, noting what's missing or underrepresented (e.g. question answering for neglected English dialects, metrics for trustworthiness). Second, we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios to the extent possible (87.5% of the time), ensuring that metrics beyond accuracy don't fall to the wayside, and that trade-offs across models and metrics are clearly exposed. We also perform 7 targeted evaluations, based on 26 targeted scenarios, to more deeply analyze specific aspects (e.g. knowledge, reasoning, memorization/copyright, disinformation). Third, we conduct a large-scale evaluation of 30 prominent language models (spanning open, limited-access, and closed models) on all 42 scenarios, including 21 scenarios that were not previously used in mainstream LM evaluation. Prior to HELM, models on average were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: now all 30 models have been densely benchmarked on a set of core scenarios and metrics under standardized conditions. Our evaluation surfaces 25 top-level findings concerning the interplay between different scenarios, metrics, and models. For full transparency, we release all raw model prompts and completions publicly 3 for further analysis, as well as a general modular toolkit for easily adding new scenarios, models, metrics, and prompting strategies. 4 We intend for HELM to be a living benchmark for the community, continuously updated with new scenarios, metrics, and models.

show abstract

Section: E5 Disinformationmentioning

confidence: 99%

Holistic Evaluation of Language Models

Liang¹,

Bommasani²,

Lee³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For our video application, we use the YouTube data by Ribeiro et al [29,41], which contains identifiers and "Up Next"-recommendations for videos from selected channels categorized to reflect different degrees and directions of radicalization. For our news application, we use subsets of the NELA-GT-2021 dataset [17], which contains 1.8 million news articles published in 2021 from 367 outlets, along with veracity labels from Media Bias/Fact Check. Prior versions of both datasets are used in the experiments reported by Fabbri et al [10].…”

Section: Setupmentioning

confidence: 99%

Reducing Exposure to Harmful Content via Graph Rewiring

Coupette,

Neumann,

Gionis

2023

Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Most media content consumed today is provided by digital platforms that aggregate input from diverse sources, where access to information is mediated by recommendation algorithms. One principal challenge in this context is dealing with content that is considered harmful. Striking a balance between competing stakeholder interests, rather than block harmful content altogether, one approach is to minimize the exposure to such content that is induced specifically by algorithmic recommendations. Hence, modeling media items and recommendations as a directed graph, we study the problem of reducing the exposure to harmful content via edge rewiring. We formalize this problem using absorbing random walks, and prove that it is NP-hard and NP-hard to approximate to within an additive error, while under realistic assumptions, the greedy method yields a (1 − 1 /𝑒)-approximation. Thus, we introduce Gamine, a fast greedy algorithm that can reduce the exposure to harmful content with or without quality constraints on recommendations. By performing just 100 rewirings on YouTube graphs with several hundred thousand edges, Gamine reduces the initial exposure by 50%, while ensuring that its recommendations are at most 5% less relevant than the original recommendations. Through extensive experiments on synthetic data and real-world data from video recommendation and news feed applications, we confirm the effectiveness, robustness, and efficiency of Gamine in practice. CCS CONCEPTS• Information systems → Recommender systems; Web applications; • Theory of computation → Random walks and Markov chains; • Mathematics of computing → Graph algorithms.

show abstract

NELA-GT-2022: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles

Cited by 2 publications

References 0 publications

Holistic Evaluation of Language Models

Holistic Evaluation of Language Models

Reducing Exposure to Harmful Content via Graph Rewiring

Contact Info

Product

Resources

About