Factuality Assessment as Modal Dependency Parsing

Abstract

As the sources of information that we consume everyday rapidly diversify, it is becoming increasingly important to develop NLP tools that help to evaluate the credibility of the information we receive. A critical step towards this goal is to determine the factuality of events in text. In this paper, we frame factuality assessment as a modal dependency parsing task that identifies the events and their sources, formally known as conceivers, and then determine the level of certainty that the sources are asserting with respect to the events. We crowdsource the first large-scale data set annotated with modal dependency structures that consists of 353 Covid-19 related news articles, 24,016 events, and 2,938 conceivers. 1 We also develop the first modal dependency parser that jointly extracts events, conceivers and constructs the modal dependency structure of a text. We evaluate the joint model against a pipeline model and demonstrate the advantage of the joint model in conceiver extraction and modal dependency structure construction when events and conceivers are automatically extracted. We believe the dataset and the models will be a valuable resource for a whole host of NLP applications such as fact checking and rumor detection.

ExcavatorCovid: Extracting Events and Relations from Text Corpora for Temporal and Causal Analysis for COVID-19

“…We present Excavator, a machine reading system that automatically constructs a Temporal and Causal Analysis Graph for COVID-19 by reading open-source text documents such as news and scientific publications. Our next steps are to integrate Modal Dependency Parsing (Yao et al, 2021) for event factuality assessment, and cross-lingual transfer learning (Nguyen et al, 2021) to make Excavator applicable to more languages.…”

Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

“…Specifically, for each candidate mention span, the algorithm computes a distribution over possible antecedent spans from the mention score (whether it is likely to be a mention) and the compatibility score of the two spans, which itself involves a feed-forward network to compute. Two more structural parsing examples in this vein are temporal dependency parsing (Ross et al, 2020) and modal dependency parsing (Yao et al, 2021). These studies approach tree building algorithmically by first performing a classification problem to identify suitable dependency pairs, then ranking them to construct a valid tree.…”

Modal Dependency Parsing via Language Model Priming

“…For instance, in Figure 1, a person is an entity but is not a conceiver as it is not the source of any event. As a result, Yao et al (2021) report relatively low conceiver extraction F-score compared to event extraction (70.4% for conceiver extraction vs. 90.8% for event extraction). Errors in conceiver extraction will propagate to the structure building stage, leading to lower overall MDS parsing accuracy.…”

“…For example, in Figure 1, our judgment of whether the event travelled has happened also crucially depends on the credibility of the information source, Jeroen Weimar, in addition to the level of certainty the information source holds towards the event. Yao et al (2021) develop the first modal dependency parser by first separately extracting events and conceivers, then building up the MDS bottomup with a ranking model. One shortcoming of this approach is that it fails to capture the fact that the status of an entity as a conceiver is conditioned on its being the information source of an event.…”

“…In this paper, we describe an approach to MDP based on language model priming in which we construct a prompt with an event and use it to predict its chain of conceivers as well as the level of certainty the conceivers hold 1 . This approach avoids the error propagation problem in Yao et al (2021) and also takes better advantage of powerful pretrained language models. Our experiments show that this approach outperforms previous models for both English and Chinese.…”