Distilling the Evidence to Augment Fact Verification Models

Portelli, Beatrice; Zhao, Jason; Schuster, Tal; Serra, Giuseppe; Santus, Enrico

doi:10.18653/v1/2020.fever-1.7

Cited by 10 publications

(6 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This, however, does not remove the threats of echo chambers and misinformation. As future work, we plan to add a new module based on our previous work [ 54 ] to better analyze phenomena related to the spread of misinformation.…”

Section: Discussionmentioning

confidence: 99%

Monitoring User Opinions and Side Effects on COVID-19 Vaccines in the Twittersphere: Infodemiology Study of Tweets

Portelli¹,

Scaboro²,

Tonino³

et al. 2022

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

Background In the current phase of the COVID-19 pandemic, we are witnessing the most massive vaccine rollout in human history. Like any other drug, vaccines may cause unexpected side effects, which need to be investigated in a timely manner to minimize harm in the population. If not properly dealt with, side effects may also impact public trust in the vaccination campaigns carried out by national governments. Objective Monitoring social media for the early identification of side effects, and understanding the public opinion on the vaccines are of paramount importance to ensure a successful and harmless rollout. The objective of this study was to create a web portal to monitor the opinion of social media users on COVID-19 vaccines, which can offer a tool for journalists, scientists, and users alike to visualize how the general public is reacting to the vaccination campaign. Methods We developed a tool to analyze the public opinion on COVID-19 vaccines from Twitter, exploiting, among other techniques, a state-of-the-art system for the identification of adverse drug events on social media; natural language processing models for sentiment analysis; statistical tools; and open-source databases to visualize the trending hashtags, news articles, and their factuality. All modules of the system are displayed through an open web portal. Results A set of 650,000 tweets was collected and analyzed in an ongoing process that was initiated in December 2020. The results of the analysis are made public on a web portal (updated daily), together with the processing tools and data. The data provide insights on public opinion about the vaccines and its change over time. For example, users show a high tendency to only share news from reliable sources when discussing COVID-19 vaccines (98% of the shared URLs). The general sentiment of Twitter users toward the vaccines is negative/neutral; however, the system is able to record fluctuations in the attitude toward specific vaccines in correspondence with specific events (eg, news about new outbreaks). The data also show how news coverage had a high impact on the set of discussed topics. To further investigate this point, we performed a more in-depth analysis of the data regarding the AstraZeneca vaccine. We observed how media coverage of blood clot–related side effects suddenly shifted the topic of public discussions regarding both the AstraZeneca and other vaccines. This became particularly evident when visualizing the most frequently discussed symptoms for the vaccines and comparing them month by month. Conclusions We present a tool connected with a web portal to monitor and display some key aspects of the public’s reaction to COVID-19 vaccines. The system also provides an overview of the opinions of the Twittersphere through graphic representations, offering a tool for the extraction of suspected adverse events from tweets with a deep learning model.

show abstract

Section: Discussionmentioning

confidence: 99%

Monitoring User Opinions and Side Effects on COVID-19 Vaccines in the Twittersphere: Infodemiology Study of Tweets

Portelli¹,

Scaboro²,

Tonino³

et al. 2022

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

show abstract

“…The rest of the documents (i.e., those with disambiguative information) are ranked and filtered out using NSNM and a threshold value. Several works (Ma et al, 2019;Nie et al, 2019b;Zhong et al, 2020;Portelli et al, 2020) 2018) aims for high precision using exact matching techniques. In addition, as we observe in Table 2, most of the works that have been developed for the competition shared task (2018) focus on hand-crafted features.…”

Section: Keyword-based Methodsmentioning

confidence: 99%

“…For the sentence retrieval task, several pipeline methods in the literature rely on the sentence retrieval component of the baseline method (Thorne et al, 2018a). Specifically, these methods (Chernyavskiy and Ilvovsky, 2019;Portelli et al, 2020;Taniguchi et al, 2018;Yin and Schütze, 2018a), use a TF-IDF vector representation along with a cosine similarity function (see Section 2.4 for a detailed description). However, there are some attempts that exploit additional representations such as the ELMo embeddings (Chakrabarty et al, 2018).…”

Section: Tf-idfmentioning

confidence: 99%

A Review on Fact Extraction and Verification

Bekoulis¹,

Papagiannopoulou²,

Deligiannis³

2020

Preprint

View full text Add to dashboard Cite

Fact Extraction and VERification (FEVER) is a recently introduced task which aims to identify the veracity of a given claim based on Wikipedia documents. A lot of methods have been proposed to address this problem which consists of the subtasks of (i) retrieving the relevant documents (and sentences) from Wikipedia and (ii) validating whether the information in the documents supports or refutes a given claim. This task is essential since it can be the building block of applications that require a deep understanding of the language such as fake news detection and medical claim verification. In this paper, we aim to get a better understanding of the challenges in the task by presenting the literature in a structured and comprehensive way. In addition, we describe the proposed methods by analyzing the technical perspectives of the different approaches and discussing the performance results on the FEVER dataset.

show abstract

“…Pre-trained BERT models have often been used for classification (supported, refuted, and not enough info). For claim verification, BERT-based models are prevalent (Soleimani et al, 2019;Portelli et al, 2020;Chernyavskiy and Ilvovsky, 2019;Nie et al, 2019;Tokala et al, 2019), while Longformer has been used for verdict prediction (Wadden et al, 2020b;Wright et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

Multi2Claim: Generating Scientific Claims from Multi-Choice Questions for Scientific Fact-Checking

Tan,

Nguyen,

Bensemann

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

In automated scientific fact-checking, machine learning models are trained to verify scientific claims given evidence. A major bottleneck of this task is the availability of large-scale training datasets on different domains, due to the required domain expertise for data annotation. However, multiple-choice question-answering datasets are readily available across many different domains, thanks to the modern online education and assessment systems. As one of the first steps towards addressing the fact-checking dataset scarcity problem in scientific domains, we propose a pipeline for automatically converting multiple-choice questions into factchecking data, which we call Multi2Claim. By applying the proposed pipeline, we generated two large-scale datasets for scientificfact-checking: Med-Fact and Gsci-Fact for the medical and general science domains, respectively. These two datasets are among the first examples of large-scale scientific-factchecking datasets. We developed baseline models for the verdict prediction task using each dataset. Additionally, we demonstrated that the datasets could be used to improve performance measured by weighted F 1 on existing fact-checking datasets such as SciFact, HEALTHVER, COVID-Fact, and CLIMATE-FEVER. In some cases, the improvement in performance was up to a 26% increase. The generated datasets are publicly available 1 .

show abstract

Distilling the Evidence to Augment Fact Verification Models

Cited by 10 publications

References 15 publications

Monitoring User Opinions and Side Effects on COVID-19 Vaccines in the Twittersphere: Infodemiology Study of Tweets

Monitoring User Opinions and Side Effects on COVID-19 Vaccines in the Twittersphere: Infodemiology Study of Tweets

A Review on Fact Extraction and Verification

Multi2Claim: Generating Scientific Claims from Multi-Choice Questions for Scientific Fact-Checking

Contact Info

Product

Resources

About