Language Modeling with Sparse Product of Sememe Experts

Abstract

Most language modeling methods rely on large-scale data to statistically learn the sequential patterns of words.In this paper, we argue that words are atomic language units but not necessarily atomic semantic units. Inspired by HowNet, we use sememes, the minimum semantic units in human languages, to represent the implicit semantics behind words for language modeling, named Sememe-Driven Language Model (SDLM). More specifically, to predict the next word, SDLM first estimates the sememe distribution given textual context. Afterwards, it regards each sememe as a distinct semantic expert, and these experts jointly identify the most probable senses and the corresponding word. In this way, SDLM enables language models to work beyond word-level manipulation to fine-grained sememe-level semantics, and offers us more powerful tools to fine-tune language models and improve the interpretability as well as the robustness of language models. Experiments on language modeling and the downstream application of headline generation demonstrate the significant effectiveness of SDLM. Source code and data used in the experiments can be accessed at https:// github.com/thunlp/SDLM-pytorch.

Modeling Semantic Compositionality with Sememe Knowledge

“…HowNet, as the most well-known sememe KB, has attracted wide research attention. Previous work applies the sememe knowledge of HowNet to various NLP applications, such as word similarity computation (Liu and Li, 2002), word sense disambiguation (Gan and Wong, 2000;Zhang et al, 2005;Duan et al, 2007), sentiment analysis (Zhu et al, 2006;Dang and Zhang, 2010;Fu et al, 2013), word representation learning (Niu et al, 2017), language modeling (Gu et al, 2018), lexicon expansion (Zeng et al, 2018) and semantic rationality evaluation .…”

“…HowNet (Dong and Dong, 2003) is a widely acknowledged sememe knowledge base (KB), which defines about 2,000 sememes and uses them to annotate over 100,000 Chinese words together with their English translations. Sememes and HowNet have been successfully utilized in a variety of NLP tasks including sentiment analysis (Dang and Zhang, 2010), word representation learning (Niu et al, 2017), language modeling (Gu et al, 2018), etc.…”

Knowledge-Augmented Language Model and Its Application to Unsupervised Named-Entity Recognition

“…Xin et al (2018) use a similarly purposed entity typing module and a LM-enhancement module. Instead of entity type generation, Gu et al (2018) propose to explicitly decompose word generation into sememe (a semantic language unit of meaning) generation and sense generation, but requires sememe labels. Yang et al (2016) propose a pointer-network LM that can point to a 1-D or 2-D database record during inference.…”

Cross-lingual Lexical Sememe Prediction

“…Since HowNet was published (Dong and Dong, 2003), it has attracted wide attention of re-searchers. Most of related works focus on applying HowNet to specific NLP tasks (Liu and Li, 2002;Zhang et al, 2005;Sun et al, 2007;Dang and Zhang, 2010;Fu et al, 2013;Niu et al, 2017;Zeng et al, 2018;Gu et al, 2018). To the best of our knowledge, only and Jin et al (2018) conduct studies of augmenting HowNet by recommending sememes for new words.…”