COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences

Singh, Shikhar; Wen, Nuan; Hou, Yu; Alipoormolabashi, Pegah; Wu, Te-Lin; Ma, Xuezhe; Peng, Nanyun

doi:10.48550/arxiv.2106.00969

Cited by 2 publications

(2 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As emphasized in a recent study (Davis 2023), the concept of commonsense reasoning implies that its involved commonsense knowledge is common. Thus, commonsense AI should be expected to generalize, that is, at least in aggregate, should not exhibit excessive performance loss across independent commonsense benchmarks, such as (Bhagavatula et al 2020;Singh et al 2021;Santos et al 2022;Kejriwal et al 2023), regardless of the specific benchmark on (the training set of) which it has been fine-tuned. In my first work (Shen and Kejriwal 2021), we evaluated this expectation by proposing a methodology and experimental study to measure the generalization ability of transformer-based language models using statistical significance analysis and a rigorous and intuitive metric (i.e., performance loss metric).…”

Section: Generalizability Evaluationmentioning

confidence: 99%

The Generalization and Robustness of Transformer-Based Language Models on Commonsense Reasoning

Shen

2024

AAAI

View full text Add to dashboard Cite

The advent of powerful transformer-based discriminative language models and, more recently, generative GPT-family models, has led to notable advancements in natural language processing (NLP), particularly in commonsense reasoning tasks. One such task is commonsense reasoning, where performance is usually evaluated through multiple-choice question-answering benchmarks. Till date, many such benchmarks have been proposed and `leaderboards' tracking state-of-the-art performance on those benchmarks suggest that transformer-based models are approaching human-like performance. However, due to documented problems such as hallucination and bias, the research focus is shifting from merely quantifying accuracy on the task to an in-depth, context-sensitive probing of LLMs' generalization and robustness. To gain deeper insight into diagnosing these models' performance in commonsense reasoning scenarios, this thesis addresses three main studies: the generalization ability of transformer-based language models on commonsense reasoning, the trend in confidence distribution of these language models confronted with ambiguous inference tasks, and a proposed risk-centric evaluation framework for both discriminative and generative language models.

show abstract

Section: Generalizability Evaluationmentioning

confidence: 99%

The Generalization and Robustness of Transformer-Based Language Models on Commonsense Reasoning

Shen

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Avicenna [1] Complete a 6000 records Crowd sourcing valid syllogism CA-EHN [84] Chinese word analogy 90,505 analogies Adapted Labelled by experts CIDER [49] Causal explanation 807 dialogues, Adapted datasets. 4539 causal triplets with expert annotations CODAH [23] Sentence completion 28,000 questions Crowd sourcing Chinese WSC [147] Pronoun resolution 1838 questions Expert construction CommonGen [86] Make a sentence 35141 concept sets Crowd sourcing from given words 77449 sentences Com2Sense [124] Is a sentence plausible? 4000 sentence pairs Crowd sourcing CosmosQA [63] Question answering 35,600 problems Crowd sourcing CommonsenseQA [130] Question answering 12,247 questions Crowd sourcing CommonsenseQA 2.0 [131] Yes/no questions 14,343 questions Gamification COPA [111] Select a conclusion causally 1000 questions Expert authors connected to a premise CREAK [104] True/False questions 13,000 questions Crowd sourcing CycIC (No paper) Question answering 10,700 questions Synthesized DefeasibleNLI [114] Does new information 250,000 examples Crowd sourcing strenghthen an inference?…”

Section: Collections Of Elementary Mathematical Word Problemsmentioning

confidence: 99%

Benchmarks for Automated Commonsense Reasoning: A Survey

Davis¹

2023

Preprint

View full text Add to dashboard Cite

More than one hundred benchmarks have been developed to test the commonsense knowledge and commonsense reasoning abilities of artificial intelligence (AI) systems. However, these benchmarks are often flawed and many aspects of common sense remain untested. Consequently, we do not currently have any reliable way of measuring to what extent existing AI systems have achieved these abilities.This paper surveys the development and uses of AI commonsense benchmarks. We discuss the nature of common sense; the role of common sense in AI; the goals served by constructing commonsense benchmarks; and desirable features of commonsense benchmarks. We analyze the common flaws in benchmarks, and we argue that it is worthwhile to invest the work needed ensure that benchmark examples are consistently high quality. We survey the various methods of constructing commonsense benchmarks. We enumerate 139 commonsense benchmarks that have been developed: 102 text-based, 18 image-based, 12 video based, and 7 simulated physical environments. We discuss the gaps in the existing benchmarks and aspects of commonsense reasoning that are not addressed in any existing benchmark. We conclude with a number of recommendations for future development of commonsense AI benchmarks.

show abstract

COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences

Cited by 2 publications

References 39 publications

The Generalization and Robustness of Transformer-Based Language Models on Commonsense Reasoning

The Generalization and Robustness of Transformer-Based Language Models on Commonsense Reasoning

Benchmarks for Automated Commonsense Reasoning: A Survey

Contact Info

Product

Resources

About