KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning

Liu, Ye; Wan, Yao; He, Lifang; Peng, Hao; Yu, Philip S.

doi:10.48550/arxiv.2009.12677

Cited by 9 publications

(12 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, the pseudo labels of items provide additional rating information, which can alleviate the cold-start issue of the data. Intuitively, this technique works as a data augmentation method [26,28,59]. We sample those pseudo items and view the difference between rounded ratings with a predicted rating as the randomness, which enhances the robustness of the local model.…”

Section: Local Differential Privacymentioning

confidence: 99%

Federated Social Recommendation with Graph Neural Network

Liu

Yang

Fan

et al. 2022

ACM Trans. Intell. Syst. Technol.

Self Cite

View full text Add to dashboard Cite

Recommender systems have become prosperous nowadays, designed to predict users’ potential interests in items by learning embeddings. Recent developments of the Graph Neural Networks (GNNs) also provide recommender systems with powerful backbones to learn embeddings from a user-item graph. However, only leveraging the user-item interactions suffers from the cold-start issue due to the difficulty in data collection. Hence, current endeavors propose fusing social information with user-item interactions to alleviate it, which is the social recommendation problem. Existing work employs GNNs to aggregate both social links and user-item interactions simultaneously. However, they all require centralized storage of the social links and item interactions of users, which leads to privacy concerns. Additionally, according to strict privacy protection under General Data Protection Regulation, centralized data storage may not be feasible in the future, urging a decentralized framework of social recommendation. As a result, we design a federated learning recommender system for the social recommendation task, which is rather challenging because of its heterogeneity, personalization, and privacy protection requirements. To this end, we devise a novel framework Fe drated So cial recommendation with G raph neural network ( FeSoG ). Firstly, FeSoG adopts relational attention and aggregation to handle heterogeneity. Secondly, FeSoG infers user embeddings using local data to retain personalization. Last but not least, the proposed model employs pseudo-labeling techniques with item sampling to protect the privacy and enhance training. Extensive experiments on three real-world datasets justify the effectiveness of FeSoG in completing social recommendation and privacy protection. We are the first work proposing a federated learning framework for social recommendation to the best of our knowledge.

show abstract

Section: Local Differential Privacymentioning

confidence: 99%

Federated Social Recommendation with Graph Neural Network

Liu

Yang

Fan

et al. 2022

ACM Trans. Intell. Syst. Technol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…KG-BART [51] first constructs a knowledge subgraph from commonsense KG, and use Glove embedding [60] based word similarity to prune irrelevant concepts. Then graph attention is employed to learn concept embeddings, which are integrated with token embedding using concept-to-subword and subword-to-concept fusion.…”

Section: Commonsense Kg Knowledge Subgraphs Are Also Very Useful For ...mentioning

confidence: 99%

“…KG-BART [51] and [44], via incorporating commonsense knowledge, show better capability on generating more natural and sensible text for the given concept set.…”

Section: Text Generationmentioning

confidence: 99%

“…Since many existing approaches incorporate entity-level knowledge, entity related tasks (e.g., entity typing and relation classification) become natural testbeds for evaluating the efficacy of these KE-PLMs. By injecting entity information Knowledge Source ERICA [66] Yes entity/relation discrimination Wikipedia, Wikidata ERNIE (THU) [108] Yes entity prediction Wikipedia/Wikidata ERNIE 2.0 (Baidu) [83] Yes masked entity/phrase N/A E-BERT [64] Yes entity/wordpiece alignment Wikipedia2Vec E(commerce)-BERT [106] Yes neighbor Product Reconstruction product graph/AutoPhrase[75] EaE [21] Yes mention detection/linking Wikipedia CokeBERT [80] Yes entity prediction Wikipedia/Wikidata COMET [6] No autoregressive ATOMIC, ConceptNet K-Adapter [96] No dependency relation Wikipedia, Wikidata, Stanford Parser KnowBERT [61] Yes entity linking WordNet, Wikipedia K-BERT [49] No finetuning WikiZh, WebtextZh, CN-DBpedia HowNet, MedicalKG KEPLER [97] Yes TransE scoring Wikipedia/Wikidata KG-BERT [102] Yes relation cross-entropy ConceptNet KG-BART [51] Yes masked concept ConceptNet KgPLM [26] Yes generative/discriminative masked entity Wikipedia/Wikidata FaE [90] Yes masked entity et.al Wikipedia/Wikidata JAKET [105] Yes entity category/relation type/masked entity Wikipedia/Wikidata LUKE [100] Yes entity prediction Wikipedia WKLM [99] Yes entity replacement detection Wikipedia/Wikidata CoLAKE [82] Yes masked entity prediction Wikipedia/Wikidata KT-NET [101] No finetuning N/A LIBERT [40] Yes lexical relation prediction WordNet SenseBERT [41] Yes supersense prediction WordNet Syntax-BERT [2] No masks induced by syntax tree parsing syntax tree SentiLARE [36] Yes POS/ word level polarity/sentiment polarity SentiWordNet [44] No finetuning ConceptNet COCOLM [104] Yes discourse relation/co-occurrence relation ASER [22] Yes autoregressive ConceptNet/ATOMIC AMS [103] Yes distractor-based loss ConceptNet GLM …”

Section: Entity-related Tasksmentioning

confidence: 99%

“…• For NLG, although PLMs are able to generate grammatically correct sentence, the generated text might not be logical or sensible [46] [51]. For example, as noted in [46], given a set of concepts {dog, frisbee, catch, throw}, GPT2 generates "A dog throws a frisbee at a football player" and T5 generates "dog catches a frisbee and throws it to a dog", neither of which aligns with human's commonsense.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey

Wei,

Wang,

Zhang

et al. 2021

Preprint

View full text Add to dashboard Cite

Pretrained Language Models (PLM) have established a new paradigm through learning informative contextualized representations on large-scale text corpus. This new paradigm has revolutionized the entire field of natural language processing, and set the new state-of-the-art performance for a wide variety of NLP tasks. However, though PLMs could store certain knowledge/facts from training corpus, their knowledge awareness is still far from satisfactory. To address this issue, integrating knowledge into PLMs have recently become a very active research area and a variety of approaches have been developed. In this paper, we provide a comprehensive survey of the literature on this emerging and fast-growing field -Knowledge Enhanced Pretrained Language Models (KE-PLMs). We introduce three taxonomies to categorize existing work. Besides, we also survey the various NLU and NLG applications on which KE-PLM has demonstrated superior performance over vanilla PLMs. Finally, we discuss challenges that face KE-PLMs and also promising directions for future research.

show abstract

DSKReG: Differentiable Sampling on Knowledge Graph for Recommendation with Relational GNN

Wang

Liu

Fan

et al. 2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

Self Cite

View full text Add to dashboard Cite

In the information explosion era, recommender systems (RSs) are widely studied and applied to discover user-preferred information. A RS performs poorly when suffering from the cold-start issue, which can be alleviated if incorporating Knowledge Graphs (KGs) as side information. However, most existing works neglect the facts that node degrees in KGs are skewed and massive amount of interactions in KGs are recommendation-irrelevant. To address these problems, in this paper, we propose Differentiable Sampling on Knowledge Graph for Recommendation with Relational GNN (DSKReG) that learns the relevance distribution of connected items from KGs and samples suitable items for recommendation following this distribution. We devise a differentiable sampling strategy, which enables the selection of relevant items to be jointly optimized with the model training procedure. The experimental results demonstrate that our model outperforms state-of-the-art KG-based recommender systems. The code is available online at https://github.com/YuWang-1024/DSKReG.

show abstract

KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning

Cited by 9 publications

References 32 publications

Federated Social Recommendation with Graph Neural Network

Federated Social Recommendation with Graph Neural Network

Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey

DSKReG: Differentiable Sampling on Knowledge Graph for Recommendation with Relational GNN

Contact Info

Product

Resources

About