Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks

Chang, Ting-Yun; Liu, Yang; Gopalakrishnan, Karthik; Hedayatnia, Behnam; Zhou, Pei; Hakkani‐Tür, Dilek

doi:10.18653/v1/2020.deelio-1.9

Cited by 21 publications

(14 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The last row in Table 3 shows the results when ensembling 5 models with majority vote, including See all Choices, Cross-Segment Attention, RoBERTa + GPT2, Multiple-Choice Datasets, and one system using external concept knowledge graph that has a classification accuracy of 79.2% [29]. This ensemble outperforms all the individual models.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Go Beyond Plain Fine-Tuning: Improving Pretrained Models for Social Commonsense

Chang

Liu

Gopalakrishnan

et al. 2021

2021 IEEE Spoken Language Technology Workshop (SLT)

Self Cite

View full text Add to dashboard Cite

Pretrained language models have demonstrated outstanding performance in many NLP tasks recently. However, their social intelligence, which requires commonsense reasoning about the current situation and mental states of others, is still developing. Towards improving language models' social intelligence, in this study we focus on the Social IQA dataset, a task requiring social and emotional commonsense reasoning. Building on top of the pretrained RoBERTa and GPT2 models, we propose several architecture variations and extensions, as well as leveraging external commonsense corpora, to optimize the model for Social IQA. Our proposed system achieves competitive results as those top-ranking models on the leaderboard. This work demonstrates the strengths of pretrained language models, and provides viable ways to improve their performance for a particular task.

show abstract

Section: Resultsmentioning

confidence: 99%

“…There are plenty of common sense resources with different focuses [1,2,3,5,26,27,28], which may be very helpful for our model to learn more broad and general concepts of our world. In [29], we investigated how to incorporate knowledge graph into the pretrained models.…”

Section: Using External Resourcesmentioning

confidence: 99%

Go Beyond Plain Fine-Tuning: Improving Pretrained Models for Social Commonsense

Chang

Liu

Gopalakrishnan

et al. 2021

2021 IEEE Spoken Language Technology Workshop (SLT)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Wang et al (2020b), for example, retrieve multi-hop knowledge paths from ConceptNet for fine-tuning LMs for multiple choice question answering. Chang et al (2020) and Bosselut et al (2021) incorporate knowledge paths from ConceptNet into pre-trained LMs for solving the SocialIQA task . However, all these approaches evaluate the effectiveness of integrating commonsense knowledge indirectly on downstream tasks, and do not explicitly evaluate the impact and relevance of knowledge for a specific system prediction.…”

Section: Related Workmentioning

confidence: 99%

Reconstructing Implicit Knowledge with Language Models

Becker

Liang²,

Frank

2021

Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Ar

View full text Add to dashboard Cite

In this work we propose an approach for generating statements that explicate implicit knowledge connecting sentences in text. We make use of pre-trained language models which we refine by fine-tuning them on specifically prepared corpora that we enriched with implicit information, and by constraining them with relevant concepts and connecting commonsense knowledge paths. Manual and automatic evaluation of the generations shows that by refining language models as proposed, we can generate coherent and grammatically sound sentences that explicate implicit knowledge which connects sentence pairs in texts -on both in-domain and out-of-domain test data.

show abstract

“…SocialIQA Task Most previous works on So-cialIQA task involve either large size of pre-trained models, and datasets (Khashabi et al, 2020;Lourie et al, 2021) or complicated models that heavily rely on external knowledge bases (Shen et al, 2020;Shwartz et al, 2020;Mitra et al, 2019;Ji et al, 2020a,b;Chang et al, 2020). Among them, Uni-fiedQA (Khashabi et al, 2020) achieved impressive performance by fine-tuning 11B T5 model (Raffel et al, 2019) with 17 existing QA datasets.…”

Section: Related Workmentioning

confidence: 99%

Semantic Categorization of Social Knowledge for Commonsense Question Answering

Wang¹,

Hou²,

Yang³

et al. 2021

Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing

View full text Add to dashboard Cite

Large pre-trained language models (PLMs) have led to great success on various commonsense question answering (QA) tasks in an end-to-end fashion. However, little attention has been paid to what commonsense knowledge is needed to deeply characterize these QA tasks. In this work, we proposed to categorize the semantics needed for these tasks using the SocialIQA as an example. Building upon our labeled social knowledge categories dataset on top of SocialIQA, we further train neural QA models to incorporate such social knowledge categories and relation information from a knowledge base. Unlike previous work, we observe our models with semantic categorizations of social knowledge can achieve comparable performance with a relatively simple model and smaller size compared to other complex approaches.

show abstract

Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks

Cited by 21 publications

References 22 publications

Go Beyond Plain Fine-Tuning: Improving Pretrained Models for Social Commonsense

Go Beyond Plain Fine-Tuning: Improving Pretrained Models for Social Commonsense

Reconstructing Implicit Knowledge with Language Models

Semantic Categorization of Social Knowledge for Commonsense Question Answering

Contact Info

Product

Resources

About