Do Nuclear Submarines Have Nuclear Captains? A Challenge Dataset for Commonsense Reasoning over Adjectives and Objects

Abstract

How do adjectives project from a noun to its parts? If a motorcycle is red, are its wheels red? Is a nuclear submarine's captain nuclear? These questions are easy for humans to judge using our commonsense understanding of the world, but are difficult for computers. To attack this challenge, we crowdsource a set of human judgments that answer the English-language question "Given a whole described by an adjective, does the adjective also describe a given part?" We build strong baselines for this task with a classification approach. Our findings indicate that, despite the recent successes of large language models on tasks aimed to assess commonsense knowledge, these models do not greatly outperform simple word-level models based on pre-trained word embeddings. This provides evidence that the amount of commonsense knowledge encoded in these language models does not extend far beyond that already baked into the word embeddings. Our dataset will serve as a useful testbed for future research in commonsense reasoning, especially as it relates to adjectives and objects.

Is “My Favorite New Movie” My Favorite Movie? Probing the Understanding of Recursive Noun Phrases

“…Other related work explores which attributes of the head noun are affected by the presence of modifiers. Mullenbach et al (2019) look at how modifiers project from a noun to its parts (e.g., does a red jeep have red tires?). Emami et al (2021) test the likelihood change of an event when a modifier is added (e.g., a false key is less likely to open a door than a key).…”

ADEPT: An Adjective-Dependent Plausibility Task

“…The Johns Hopkins Ordinal Commonsense Inference (JOCI) task generalizes RTE to the problem of determining relative change in semantic plausibility on an ordinal 5-level Likert scale (from impossible to very likely) (Zhang et al, 2017). Other semantic plausibility datasets have collected judgments for the plausibility of single events (Wang et al, 2018b) and the plausibility of adjectives modifying a meronym (Mullenbach et al, 2019). Such plausibility tasks have often been solved using either data-driven methods (Huang and Luo, 2017;Sasaki et al, 2017) or pre-trained LMs .…”

Is "My Favorite New Movie" My Favorite Movie? Probing the Understanding of Recursive Noun Phrases

“…Other recent work also explored which attributes of the head noun are affected by the presence of modifiers. Mullenbach et al (2019) looked at how modifiers project from a noun to its parts (e.g., does a red jeep have red tires?). Emami et al (2021) tested the likelihood change of an event when a modifier is added (e.g., a false key is less likely to open a door than a key.).…”