ADEPT: An Adjective-Dependent Plausibility Task

Abstract

A false contract is more likely to be rejected than a contract is, yet a false key is less likely than a key to open doors. While correctly interpreting and assessing the effects of such adjective-noun pairs (e.g., false key) on the plausibility of given events (e.g., opening doors) underpins many natural language understanding tasks, doing so often requires a significant degree of world knowledge and common-sense reasoning. We introduce ADEPT -a large-scale semantic plausibility task consisting of over 16 thousand sentences that are paired with slightly modified versions obtained by adding an adjective to a noun. Overall, we find that while the task appears easier for human judges (85% accuracy), it proves more difficult for transformer-based models like RoBERTa (71% accuracy). Our experiments also show that neither the adjective itself nor its taxonomic class suffice in determining the correct plausibility judgement, emphasizing the importance of endowing automatic natural language understanding systems with more context sensitivity and commonsense reasoning.

Is “My Favorite New Movie” My Favorite Movie? Probing the Understanding of Recursive Noun Phrases

“…Mullenbach et al (2019) look at how modifiers project from a noun to its parts (e.g., does a red jeep have red tires?). Emami et al (2021) test the likelihood change of an event when a modifier is added (e.g., a false key is less likely to open a door than a key). Apidianaki and Garí Soler (2021) study the prototypical properties of nouns (e.g., a strawberry entails a red strawberry).…”

“…(2) MPE (Lai et al, 2017) for our MPTE; and (3) ADEPT (Emami et al, 2021) for our EPC. We choose SOTA and close-to-SOTA models on these benchmarks as probing candidates, including BERT (Devlin et al, 2019), RoBERTa (Liu et al, 2019b), BART (Lewis et al, 2020), and GPT3 (Brown et al, 2020).…”

Is "My Favorite New Movie" My Favorite Movie? Probing the Understanding of Recursive Noun Phrases

“…Mullenbach et al (2019) looked at how modifiers project from a noun to its parts (e.g., does a red jeep have red tires?). Emami et al (2021) tested the likelihood change of an event when a modifier is added (e.g., a false key is less likely to open a door than a key.). Apidianaki and Garí Soler (2021) studied the prototypical properties of nouns (e.g., a strawberry entails a red strawberry).…”

“…Event Plausibility Comparison (EPC) follows the task formalization by Emami et al (2021). Given two events, Event1 and Event2, a model needs to assess the plausibility of Event2 compared to that of Event1.…”

“…(2) MPE (Lai et al, 2017) for our MPTE; and (3) ADEPT (Emami et al, 2021) for our EPC. We choose SOTA and close-to-SOTA models on existing benchmarks as the candidates for probing, including BERT (Devlin et al, 2019), RoBERTa (Liu et al, 2019b), BART (Lewis et al, 2020), and GPT3 (Brown et al, 2020), in different sizes.…”