Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding

Storks, Shane; Gao, Qiaozi; Zhang, Yichi; Chai, Joyce

doi:10.18653/v1/2021.findings-emnlp.422

Cited by 7 publications

(13 citation statements)

References 52 publications

(24 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Datasets such as Propara (Tandon et al, 2020), Recipes , Procedural Cyber-Security text (Pal et al, 2021), and OpenPI (Tandon et al, 2020) are in the same direction. Procedural reasoning can also be influential in addressing causal reasoning (WIQA) , story understanding (Trip) (Storks et al, 2021), and abstractive multi-modal question answering (RecipeQA) (Yagcioglu et al, 2018).…”

Section: Related Researchmentioning

confidence: 99%

“…Procedural reasoning is the ability to track entities and understand their evolution given a sequence of actions (Tandon et al, 2020). This kind of reasoning is crucial in understanding recipes Yagcioglu et al, 2018), manuals and tutorials (Tandon et al, 2020;Wu et al, 2022), cybersecurity text (Pal et al, 2021), natural events (Tandon et al, 2020), and even stories (Storks et al, 2021). An example of a procedural text in the natural event domain, its entities of interest, and their state changes are shown in Figure 1.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Role of Semantic Parsing in Understanding Procedural Text

Faghihi¹,

Kordjamshidi²,

Teng³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper, we investigate whether symbolic semantic representations, extracted from deep semantic parsers, can help reasoning over the states of involved entities in a procedural text. We consider a deep semantic parser (TRIPS) and semantic role labeling as two sources of semantic parsing knowledge. First, we propose PROPOLIS, a symbolic parsing-based procedural reasoning framework. Second, we integrate semantic parsing information into stateof-the-art neural models to conduct procedural reasoning. Our experiments indicate that explicitly incorporating such semantic knowledge improves procedural understanding. This paper presents new metrics for evaluating procedural reasoning tasks that clarify the challenges and identify differences among neural, symbolic, and integrated models.

show abstract

Section: Related Researchmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The Role of Semantic Parsing in Understanding Procedural Text

Faghihi¹,

Kordjamshidi²,

Teng³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Many methods have been proposed to solve the NLI problem. These approaches change over time, with the development of new methods and models being introduced, and can currently be divided into 3 main groups: symbolic (logic), statistical, and neural networks (deep learning) [9]. In the next part of this section, we will survey some approaches that have been applied to the NLI problem…”

Section: Approaches For Nlimentioning

confidence: 99%

“…Symbolic approaches use logical forms and processes to make inferences [9]. These approaches have primarily been applied in the early NLI challenges.…”

Section: Symbolic Approachesmentioning

confidence: 99%

VLSP 2021 - vnNLI Challenge: Vietnamese and English-Vietnamese Textual Entailment

Ngo

Hoang

Nguyen

et al. 2022

JCSCE

View full text Add to dashboard Cite

This paper presents the first challenge on recognizing textual entailment (RTE), also known as natural language inference (NLI), held in a Vietnamese Language and Speech Processing workshop (VLSP 2021).The challenge aims to determine, for a given pair of sentences, whether the two sentences semantically agree, disagree, or are neutral/irrelevant to each other. The input sentences are in English or Vietnamese and may not be in the same language. This task is important in identifying, from different information sources, the evidence that supports or refutes a statement. The identification of such evidence is subsequently useful for many information tracking applications, such as opinion mining, brand and reputation management, and particularly fighting against fake news.Through this challenge, we would like to provide an opportunity for participants who are interested in the problem, to contribute their knowledge to improve the existing techniques and methods for the task, so as to enhance the effectiveness of those applications.In the paper, we introduce a collection of Vietnamese and English sentences in the domain of health that we built to serve as a benchmarking dataset for the task. We also describe the evaluation results of systems participating in the challenge.

show abstract

“…Commonsense reasoning has become a resurgent area of research in both the NLP and broader AI communities 1 . (Davis, 2014; Storks et al, 2019; Zang et al, 2013), despite having been introduced as an early AI challenge more than 50 years ago, in the context of machine translation (Bar‐Hillel, 1960). Traditionally, it was believed that the problem could only be solved through a combination of techniques, including Web mining, logical reasoning, handcrafted knowledge bases and crowdsourcing (Davis & Marcus, 2015; H. Liu & Singh, 2004; Moore, 1982).…”

Section: Introductionmentioning

confidence: 99%

An experimental study measuring the generalization of fine‐tuned language representation models across commonsense reasoning benchmarks

Shen

Kejriwal

2023

Expert Systems

View full text Add to dashboard Cite

In the last 5 years, language representation models, such as BERT and GPT-3, based on transformer neural networks, have led to enormous progress in natural language processing (NLP). One such NLP task is commonsense reasoning, where performance is usually evaluated through multiple-choice question answering benchmarks. Till date, many such benchmarks have been proposed, and 'leaderboards' tracking stateof-the-art performance on those benchmarks suggest that transformer-based models are approaching human-like performance. Because these are commonsense benchmarks, however, such a model should be expected to generalize, that is, at least in aggregate, should not exhibit excessive performance loss across independent commonsense benchmarks regardless of the specific benchmark on (the training set of) which it has been fine-tuned. In this article, we evaluate this expectation by proposing a methodology and experimental study to measure the generalization ability of language representation models using a rigorous and intuitive metric. Using five established commonsense reasoning benchmarks, our experimental study shows that the models do not generalize well, and may be (potentially) susceptible to issues such as dataset bias. The results therefore suggest that current performance on benchmarks may be an over-estimate, especially if we want to use such models on novel commonsense problems for which a 'training' dataset may not be available, for the language representation model, to fine-tune on.

show abstract

Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding

Cited by 7 publications

References 52 publications

The Role of Semantic Parsing in Understanding Procedural Text

The Role of Semantic Parsing in Understanding Procedural Text

VLSP 2021 - vnNLI Challenge: Vietnamese and English-Vietnamese Textual Entailment

An experimental study measuring the generalization of fine‐tuned language representation models across commonsense reasoning benchmarks

Contact Info

Product

Resources

About