FAKTA: An Automatic End-to-End Fact Checking System

Nadeem, Moin; Fang, Wei; Xu, Brian; Mohtarami, Mitra; Glass, James

doi:10.48550/arxiv.1906.04164

Cited by 3 publications

(6 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Algorithmic misinformation detection can be composed of many sub-tasks, which some systems tackle independently while others attempt to solve in an end-to-end fashion. While the specifics of these tasks may evolve and change over time, we draw from Guo et al [34] to differentiate between three core (sequential) tasks: (1) Check-worthiness, which aims to spot factual claims that are worthy of fact-checking [11,31,39,45], (2) Evidence retrieval of potential evidence for identified claims [21,49,56,60,66,70,74] , and (3) verdict prediction, which aims to establish the veracity of a claim [60,63,74]. In a survey on the topic by Zhou and Zafarani [78], the authors identify how misinformation can be detected from four perspectives: (1) the false knowledge it carries; (2) its writing style; (3) its propagation patterns; and (4) the credibility of its source.…”

Section: Algorithmic Misinformation Detectionmentioning

confidence: 99%

“…First, some methods (such as FAKTA [60]) utilize Google's commercial API to automatically search claims and retrieve (potentially) relevant evidence from the broader internet corpus. In some instances, this is followed by a post-processing steps in which results retrieved from search engines are merged with a database measuring the credibility of sources of information, and only evidence from sources deemed credible are kept [60]. The second type of approach is especially tailored to curb the spread of misinformation that has been previously fact-checked by humans.…”

Section: Evidence Retrievalmentioning

confidence: 99%

“…Measurement bias may also emerge when relying on crowdsourced indicators of credibility. For example, FAKTA [60] incorporates metadata (𝐶 𝑚𝑒𝑡𝑎 ) regarding the credibility of sources of evidence, which is ultimately used to algorithmically weight the stance of evidence towards a claim. The metadata is pulled from an existing, crowd-sourced database 4 that contains ratings on roughly 4,300 sources of evidence.…”

Section: Sources Of Harmsmentioning

confidence: 99%

“…The predicted stance can then be viewed as metadata in relation to each evidence document, serving as input to a verdict generation algorithm. For example, FAKTA's end-to-end automated fact-detection system uses the stance and credibility of supporting pieces of evidence to yield a final prediction for the veracity of a claim [60].…”

Section: Automated Verdict Generationmentioning

confidence: 99%

See 3 more Smart Citations

Justice in Misinformation Detection Systems: An Analysis of Algorithms, Stakeholders, and Potential Harms

Neumann,

De-Arteaga,

Fazelpour

2022

Preprint

View full text Add to dashboard Cite

Faced with the scale and surge of misinformation on social media, many platforms and fact-checking organizations have turned to algorithms for automating key parts of misinformation detection pipelines. While offering a promising solution to the challenge of scale, the ethical and societal risks associated with algorithmic misinformation detection are not well-understood. In this paper, we employ and extend upon the notion of informational justice to develop a framework for explicating issues of justice relating to representation, participation, distribution of benefits and burdens, and credibility in the misinformation detection pipeline. Drawing on the framework: (1) we show how injustices materialize for stakeholders across three algorithmic stages in the pipeline; (2) we suggest empirical measures for assessing these injustices; and (3) we identify potential sources of these harms. This framework should help researchers, policymakers, and practitioners reason about potential harms or risks associated with these algorithms and provide conceptual guidance for the design of algorithmic fairness audits in this domain.

show abstract

Section: Algorithmic Misinformation Detectionmentioning

confidence: 99%

Section: Evidence Retrievalmentioning

confidence: 99%

Section: Sources Of Harmsmentioning

confidence: 99%

Section: Automated Verdict Generationmentioning

confidence: 99%

See 2 more Smart Citations

Justice in Misinformation Detection Systems: An Analysis of Algorithms, Stakeholders, and Potential Harms

Neumann,

De-Arteaga,

Fazelpour

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…With a fast-paced modern news cycle, many journalists and fact-checkers are under increased stress to be more efficient in their daily work. To assist in this process, automated fact-checking has been proposed as a potential solution [3,4,5,6,7]. Automated fact-checking systems aim to assess the veracity of claims through the collection and assessment of news articles and other relevant documents pertaining to the claim at hand.…”

Section: Introductionmentioning

confidence: 99%

Taking a Stance on Fake News: Towards Automatic Disinformation Assessment via Deep Bidirectional Transformer Language Models for Stance Detection

Dulhanty,

Deglint,

Daya

et al. 2019

Preprint

View full text Add to dashboard Cite

The exponential rise of social media and digital news in the past decade has had the unfortunate consequence of escalating what the United Nations has called a global topic of concern: the growing prevalence of disinformation 1 . Given the complexity and time-consuming nature of combating disinformation through human assessment, one is motivated to explore harnessing AI solutions to automatically assess news articles for the presence of disinformation. A valuable first step towards automatic identification of disinformation is stance detection, where given a claim and a news article, the aim is to predict if the article agrees, disagrees, takes no position, or is unrelated to the claim. Existing approaches in literature have largely relied on hand-engineered features or shallow learned representations (e.g., word embeddings) to encode the claim-article pairs, which can limit the level of representational expressiveness needed to tackle the high complexity of disinformation identification. In this work, we explore the notion of harnessing large-scale deep bidirectional transformer language models for encoding claim-article pairs in an effort to construct state-of-the-art stance detection geared for identifying disinformation. Taking advantage of bidirectional cross-attention between claim-article pairs via pair encoding with self-attention, we construct a large-scale language model for stance detection by performing transfer learning on a RoBERTa deep bidirectional transformer language model, and were able to achieve state-of-the-art performance (weighted accuracy of 90.01%) on the Fake News Challenge Stage 1 (FNC-I) benchmark. These promising results serve as motivation for harnessing such large-scale language models as powerful building blocks for creating effective AI solutions to combat disinformation.

show abstract

Coreference Resolution for Improving Performance Measures of Classification Tasks

Šteflovič,

Kapusta

2023

Applied Sciences

View full text Add to dashboard Cite

There are several possibilities to improve classification in natural language processing tasks. In this article, we focused on the issue of coreference resolution that was applied to a manually annotated dataset of true and fake news. This dataset was used for the classification task of fake news detection. The research aimed to determine whether performing coreference resolution on the input data before classification or classifying them without performing coreference resolution is more effective. We also wanted to verify whether it is possible to enhance classifier performance metrics by incorporating coreference resolution into the data preparation process. A methodology was proposed, in which we described the implementation methods in detail, starting from the identification of entity mentions in the text using the neuralcoref algorithm, then through word-embedding models (TF–IDF, Doc2Vec), and finally to several machine learning methods. The result was a comparison of the implemented classifiers based on the performance metrics described in the theoretical part. The best result for accuracy was observed for the dataset with coreference resolution applied, which had a median value of 0.8149, while for the F1 score, the best result had a median value of 0.8101. However, the more important finding is that the processed data with the application of coreference resolution led to an improvement in performance metrics in the classification tasks.

show abstract

FAKTA: An Automatic End-to-End Fact Checking System

Cited by 3 publications

References 5 publications

Justice in Misinformation Detection Systems: An Analysis of Algorithms, Stakeholders, and Potential Harms

Justice in Misinformation Detection Systems: An Analysis of Algorithms, Stakeholders, and Potential Harms

Taking a Stance on Fake News: Towards Automatic Disinformation Assessment via Deep Bidirectional Transformer Language Models for Stance Detection

Coreference Resolution for Improving Performance Measures of Classification Tasks

Contact Info

Product

Resources

About