Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey

Bhargava, Prajjwal; Ng, Vincent

doi:10.1609/aaai.v36i11.21496

Cited by 16 publications

(9 citation statements)

References 48 publications

(61 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To represent words or sentences with vectors, a pre-trained model is welcomed due to its high-dimensional space and semantic representation. Such models have been widely accepted by both academic and industrial researchers in the past ten years [30,33,34]. They are pre-trained on an original task with a large corpus and used on a target task by tuning the corresponding parameters according to the characteristics of the target task.…”

Section: Word Embeddingmentioning

confidence: 99%

An Embedding-Based Approach to Repairing OWL Ontologies

Yang

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

High-quality ontologies are critical to ontology-based applications, such as natural language understanding and information extraction, but logical conflicts naturally occur in the lifecycle of ontology development. To deal with such conflicts, conflict detection and ontology repair become two critical tasks, and we focus on repairing ontologies. Most existing approaches for ontology repair rely on the syntax of axioms or logical consequences but ignore the semantics of axioms. In this paper, we propose an embedding-based approach by considering sentence embeddings of axioms, which translates axioms into semantic vectors and provides facilities to compute semantic similarities among axioms. A threshold-based algorithm and a signature-based algorithm are designed to repair ontologies with the help of detected conflicts and axiom embeddings. In the experiments, our proposed algorithms are compared with existing ones over 20 real-life incoherent ontologies. The threshold-based algorithm with different distance metrics is further evaluated with 10 distinct thresholds and 3 pre-trained models. The experimental results show that the embedding-based algorithms could achieve promising performances.

show abstract

Section: Word Embeddingmentioning

confidence: 99%

An Embedding-Based Approach to Repairing OWL Ontologies

Yang

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…It is widely acknowledged that LLMs, trained on a huge amount of data, are able to obtain broad knowledge covering a wide range of domains (Rae et al 2021;Hoffmann et al 2022;Touvron et al 2023;Du et al 2022a;Guo et al 2023), including commonsense knowledge (West et al 2022;Bian et al 2023;Bang et al 2023). However, commonsense reasoning is still regarded as a major challenge for LLMs (Zhou et al 2020;Bhargava and Ng 2022). Studies disclose that LLMs fall short in performing adequate commonsense reasoning (Wei et al 2022).…”

Section: Introductionmentioning

confidence: 99%

CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models

Shi,

You,

Huang

et al. 2024

AAAI

View full text Add to dashboard Cite

As an indispensable ingredient of intelligence, commonsense reasoning is crucial for large language models (LLMs) in real-world scenarios. In this paper, we propose CORECODE, a dataset that contains abundant commonsense knowledge manually annotated on dyadic dialogues, to evaluate the commonsense reasoning and commonsense conflict detection capabilities of Chinese LLMs. We categorize commonsense knowledge in everyday conversations into three dimensions: entity, event, and social interaction. For easy and consistent annotation, we standardize the form of commonsense knowledge annotation in open-domain dialogues as "domain: slot = value". A total of 9 domains and 37 slots are defined to capture diverse commonsense knowledge. With these pre-defined domains and slots, we collect 76,787 commonsense knowledge annotations from 19,700 dialogues through crowdsourcing. To evaluate and enhance the commonsense reasoning capability for LLMs on the curated dataset, we establish a series of dialogue-level reasoning and detection tasks, including commonsense knowledge filling, commonsense knowledge generation, commonsense conflict phrase detection, domain identification, slot identification, and event causal inference. A wide variety of existing open-source Chinese LLMs are evaluated with these tasks on our dataset. Experimental results demonstrate that these models are not competent to predict CORECODE's plentiful reasoning content, and even ChatGPT could only achieve 0.275 and 0.084 accuracy on the domain identification and slot identification tasks under the zero-shot setting. We release the data and codes of CORECODE at https://github.com/danshi777/CORECODE to promote commonsense reasoning evaluation and study of LLMs in the context of daily conversations.

show abstract

“…by failing to account for the figurative interpretation of the IE (Balahur et al, 2010). This study focuses on injecting IE-related knowledge into small-frame PTLMs known for their wide use, such as BERT (Devlin et al, 2019) and BART (Lewis et al, 2020), considering their struggle to understand the figurative meanings of IEs (Bhargava and Ng, 2022;Zeng and Bhat, 2022). We discuss the corresponding capabilities of large PTLMs, such as GPT-3.5, in the limitation section.…”

Section: Introductionmentioning

confidence: 99%

“…We rely on psycholinguistic findings about the impact of IE-related aspects, such as mental states, emotions, and likely actions, on human IE comprehension (Rohani et al, 2012;Saban-Bezalel and Mashal, 2019), to explore the use of commonsense knowledge about IEs towards their comprehension. Specifically, we build on the findings that commonsense knowledge graphs (KGs), e.g., ATOMIC 20 20 (Hwang et al, 2021), organized as if-then relations for inferential knowledge enable linguistic and social reasoning abilities for PTLMs (Bhargava and Ng, 2022). Indeed, models relying on their applications have benefited figurative language processing, such as their interpretation (Chakrabarty et al, 2022a) and generation (Chakrabarty et al, 2021b).…”

Section: Introductionmentioning

confidence: 99%

IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions

Zeng,

Cheng,

Nanniyur

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Idiomatic expression (IE) processing and comprehension have challenged pre-trained language models (PTLMs) because their meanings are non-compositional. Unlike prior works that enable IE comprehension through finetuning PTLMs with sentences containing IEs, in this work, we construct IEKG, a commonsense knowledge graph for figurative interpretations of IEs. This extends the established ATOMIC 20 20 (Hwang et al., 2021) graph, converting PTLMs into knowledge models (KMs) that encode and infer commonsense knowledge related to IE use. Experiments show that various PTLMs can be converted into KMs with IEKG. We verify the quality of IEKG and the ability of the trained KMs with automatic and human evaluation. Through applications in natural language understanding, we show that a PTLM injected with knowledge from IEKG exhibits improved IE comprehension ability and can generalize to IEs unseen during training.

show abstract

Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey

Cited by 16 publications

References 48 publications

An Embedding-Based Approach to Repairing OWL Ontologies

An Embedding-Based Approach to Repairing OWL Ontologies

CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models

IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions

Contact Info

Product

Resources

About