CSE: Conceptual Sentence Embeddings based on Attention Model

Abstract

Most sentence embedding models typically represent each sentence only using word surface, which makes these models indiscriminative for ubiquitous homonymy and polysemy. In order to enhance representation capability of sentence, we employ conceptualization model to assign associated concepts for each sentence in the text corpus, and then learn conceptual sentence embedding (CSE). Hence, this semantic representation is more expressive than some widely-used text representation models such as latent topic model, especially for short-text. Moreover, we further extend CSE models by utilizing a local attention-based model that select relevant words within the context to make more efficient prediction. In the experiments, we evaluate the CSE models on two tasks, text classification and information retrieval. The experimental results show that the proposed models outperform typical sentence embedding models.

Self-Attentive, Multi-Context One-Class Classification for Unsupervised Anomaly Detection on Text

“…Wellknown examples include word2vec (Mikolov et al, 2013), GloVe (Pennington et al, 2014), or fastText . Approaches for learning sentence embeddings have also been introduced, including SkipThought (Kiros et al, 2015), ParagraphVector (Le and Mikolov, 2014), Conceptual Sentence Embedding (Wang et al, 2016), Sequential Denoising Autoencoders (Hill et al, 2016) or Fast-Sent (Hill et al, 2016). In a comparison of unsupervised sentence embedding models, Hill et al (2016) show that the optimal embedding critically depends on the targeted downstream task.…”

Bridging Text Visualization and Mining: A Task-Driven Survey

“…Trends over time between visualization and data mining are revealed through spark lines appearing beside the concept label. Since the total number varies widely across concepts, we normalized the spark lines for each concept so that they reveal the relative number of papers [327], [335], [339] word/phrase/entity-level (679) [4], [286], [289] document-level (288) [14], [252], [376] hybrid (387) [40], [155], [349] model inference (1335) non-probabilistic inference (267) [91], [101], [336] probabilistic inference (1160) [185], [198], [360] modeling (3085) models for classification (1636) [45], [176], [216] models for clustering (908) [300], [301], [361] models for dimension reduction (247) [129], [221], [287] topic models (1089) [32], [33], [139] models for regression (256) [49], [149], [298] language model (271) [24], [89], [158] graphical models (187) [262], [268], [367] neural networks (412) [175], [193], [224] mixture models (128)…”

Leveraging Concept-Enhanced Pre-Training Model and Masked-Entity Language Model for Named Entity Disambiguation

“…E.g., [19] introduced an entity-level masking strategy to ensure that all of the words in the same entity were masked during word representation training, instead of only one word or character being masked; [20] updated contextual word representations via a form of wordto-entity attention, by inserting prior knowledge into a deep neural model. On the other hand, previous work has demonstrated that, leveraging extra lexical knowledge (e.g., concept etc.,) can significantly boost the efficiency of contextualized embeddings for word [15], entity [21], relation [22], sentence [23], and so on. Overall, the lexicon-enhanced contextualized embedding representation produced by these models has produced substantial gains in a number of downstream NLP tasks.…”