NOPE: A Corpus of Naturally-Occurring Presuppositions in English

Parrish, Alicia; Schuster, Sebastian; Warstadt, Alex; Agha, Omar; Lee, Soo‐Hwan; Zhao, Zhuoye; Bowman, Samuel R.; Linzen, Tal

doi:10.18653/v1/2021.conll-1.28

Cited by 9 publications

(17 citation statements)

References 53 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A different approach to the data source is presented by for example (Jeretic et al 2020), in which sentences are semi-automatically generated. In contrast to datasets such as (Parrish et al 2021), we did not make any changes to the source material. The only changes we have made relate to the extended version of the dataset (see Section 3.2.8).…”

Section: Data Sourcementioning

confidence: 99%

“…Many datasets deliberately include several linguistic phenomena, for example several types of presupposition, such as (Parrish et al 2021) or distinguishes many different signatures of main verbs, for example (Ross and Pavlick 2019b) p . A similar approach is in (Rudinger, White, and Van Durme 2018), that is factive verbs are one of several types of expressions that are of interest.…”

Section: Our Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Polish natural language inference and factivity: An expert-based dataset and benchmarks

2023

View full text Add to dashboard Cite

Despite recent breakthroughs in Machine Learning for Natural Language Processing, the Natural Language Inference (NLI) problems still constitute a challenge. To this purpose, we contribute a new dataset that focuses exclusively on the factivity phenomenon; however, our task remains the same as other NLI tasks, that is prediction of entailment, contradiction, or neutral (ECN). In this paper, we describe the LingFeatured NLI corpus and present the results of analyses designed to characterize the factivity/non-factivity opposition in natural language. The dataset contains entirely natural language utterances in Polish and gathers 2432 verb-complement pairs and 309 unique verbs. The dataset is based on the National Corpus of Polish (NKJP) and is a representative subcorpus in regard to syntactic construction [V][że][cc]. We also present an extended version of the set (3035 sentences) consisting more sentences with internal negations. We prepared deep learning benchmarks for both sets. We found that transformer BERT-based models working on sentences obtained relatively good results ( $\approx 89\%$ F1 score on base dataset). Even though better results were achieved using linguistic features ( $\approx 91\%$ F1 score on base dataset), this model requires more human labor (humans in the loop) because features were prepared manually by expert linguists. BERT-based models consuming only the input sentences show that they capture most of the complexity of NLI/factivity. Complex cases in the phenomenon—for example, cases with entitlement (E) and non-factive verbs—still remain an open issue for further research.

show abstract

Section: Data Sourcementioning

confidence: 99%

Section: Our Datasetmentioning

confidence: 99%

Polish natural language inference and factivity: An expert-based dataset and benchmarks

2023

View full text Add to dashboard Cite

show abstract

“…We release our dataset, IDIOMEM, to facilitate future research on memorization recall in LMs. Diagnostic datasets, such as IDIOMEM, have often proven useful in the past (Sugawara et al, 2022;Parrish et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Understanding Transformer Memorization Recall Through Idioms

Haviv,

Cohen,

Gidron

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

To produce accurate predictions, language models (LMs) must balance between generalization and memorization. Yet, little is known about the mechanism by which transformer LMs employ their memorization capacity. When does a model decide to output a memorized phrase, and how is this phrase then retrieved from memory? In this work, we offer the first methodological framework for probing and characterizing recall of memorized sequences in transformer LMs. First, we lay out criteria for detecting model inputs that trigger memory recall, and propose idioms as inputs that typically fulfill these criteria. Next, we construct a dataset of English idioms and use it to compare model behavior on memorized vs. non-memorized inputs. Specifically, we analyze the internal prediction construction process by interpreting the model's hidden representations as a gradual refinement of the output probability distribution. We find that across different model sizes and architectures, memorized predictions are a two-step process: early layers promote the predicted token to the top of the output distribution, and upper layers increase model confidence. This suggests that memorized information is stored and retrieved in the early layers of the network. Last, we demonstrate the utility of our methodology beyond idioms in memorized factual statements. Overall, our work makes a first step towards understanding memory recall, and provides a methodological basis for future studies of transformer memorization. 1

show abstract

“…(QA) 2 also has topical connections to implicit reasoning in QA (de Marneffe et al, 2009;Louis et al, 2020;Geva et al, 2021), fact verification in QA (Chen et al, 2021;Park et al, 2022), and evaluation work in other domains of NLP for non-atissue meaning (Clausen and Manning, 2009;Tremper and Frank, 2011;Cianflone et al, 2018;Kim et al, 2019;Jeretic et al, 2020;Parrish et al, 2021;Jiang and de Marneffe, 2021, among many others). The literature on cooperative response generation is also more broadly relevant (Kaplan, 1982;Wahlster et al, 1983;Joshi et al, 1984;Gaasterland et al, 1992).…”

Section: Related Workmentioning

confidence: 99%

(QA)2: Question Answering with Questionable Assumptions: Question Answering with Questionable Assumptions

Kim¹,

Htut²,

Bowman³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Naturally occurring information-seeking questions often contain questionable assumptions-assumptions that are false or unverifiable. Questions containing questionable assumptions are challenging because they require a distinct answer strategy that deviates from typical answers for information-seeking questions. For instance, the question When did Marie Curie discover Uranium? cannot be answered as a typical when question without addressing the false assumption Marie Curie discovered Uranium. In this work, we propose (QA) 2 (Question Answering with Questionable Assumptions), an open-domain evaluation dataset consisting of naturally occurring search engine queries that may or may not contain questionable assumptions. To be successful on (QA) 2 , systems must be able to detect questionable assumptions and also be able to produce adequate responses for both typical information-seeking questions and ones with questionable assumptions. Through human rater acceptability on end-to-end QA with (QA) 2 , we find that current models do struggle with handling questionable assumptions, leaving substantial headroom for progress. * Equal contribution, corresponding authors ∆ Work partly done at NYU before joining BU. δ Work done at NYU before joining Amazon. 1 We use the term questionable assumptions instead of presupposition failure to capture failures of both true presup-

show abstract

NOPE: A Corpus of Naturally-Occurring Presuppositions in English

Cited by 9 publications

References 53 publications

Polish natural language inference and factivity: An expert-based dataset and benchmarks

Polish natural language inference and factivity: An expert-based dataset and benchmarks

Understanding Transformer Memorization Recall Through Idioms

(QA)2: Question Answering with Questionable Assumptions: Question Answering with Questionable Assumptions

Contact Info

Product

Resources

About