Subword-level Composition Functions for Learning Word Embeddings

Li, Bofang; Drozd, Aleksandr; Liu, Tao; Du, Xiaoyong

doi:10.18653/v1/w18-1205

Cited by 25 publications

(26 citation statements)

References 34 publications

(28 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, the subword embedding sequence is passed to a composition function, which computes the final word representation. Li et al (2018) and Zhu et al (2019) have empirically verified that composition by simple addition, among other more complex composition functions, is a robust choice. Therefore, we use addition in all our experiments.…”

Section: Methodsmentioning

confidence: 98%

On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages

Zhu

Heinzerling

Vulić

et al. 2019

Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

Recent work has validated the importance of subword information for word representation learning. Since subwords increase parameter sharing ability in neural models, their value should be even more pronounced in low-data regimes. In this work, we therefore provide a comprehensive analysis focused on the usefulness of subwords for word representation learning in truly low-resource scenarios and for three representative morphological tasks: fine-grained entity typing, morphological tagging, and named entity recognition. We conduct a systematic study that spans several dimensions of comparison: 1) type of data scarcity which can stem from the lack of taskspecific training data, or even from the lack of unannotated data required to train word embeddings, or both; 2) language type by working with a sample of 16 typologically diverse languages including some truly low-resource ones (e.g. Rusyn, Buryat, and Zulu); 3) the choice of the subword-informed word representation method. Our main results show that subword-informed models are universally useful across all language types, with large gains over subword-agnostic embeddings. They also suggest that the effective use of subwords largely depends on the language (type) and the task at hand, as well as on the amount of available data for training the embeddings and taskbased models, where having sufficient in-task data is a more critical requirement.

show abstract

Section: Methodsmentioning

confidence: 98%

On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages

Zhu

Heinzerling

Vulić

et al. 2019

Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

show abstract

“…The proposed embeddings -Phonetic CNN subword embeddings -comprise of two distinct embeddings, concatenated together. First, each word w is processed using the CNN-based subword-level embedding model [10], which results in word embedding W for each word. Second, each word is also processed using the phonetic level model, as presented in [40], which gives the phonetic level embedding of the word P ho_w.…”

Section: Phonetic Cnn Subword Embeddingsmentioning

confidence: 99%

“…Li and others [10] proposed the CNN-based subword-level composition function for learning word embeddings. CNN-based subword-level embedding model (Fig.…”

Section: Cnn-based Subword-level Embedding Modelmentioning

confidence: 99%

“…CNN-based subword-level embedding model (Fig. 2b) is able to distinguish different derivation types, with the advantage of the former better than FastText subword-level embedding model [10,56] is a fully-connected layer with the number of units corresponding to the desired size of embedding. The output of the fully-connected layer is used for predicting contextual words, by using negative sampling [61], a standard practice to optimize algorithm performance.…”

Section: Cnn-based Subword-level Embedding Modelmentioning

confidence: 99%

“…FastText subword embeddings which are pioneered by Bojanowski and others [50] and used by [35] for generating Arabic poetry, (2.) Phonetic CNN subword embeddings constructed by using two models: the CNN-based subword-level embedding model [10], and the phonetic-level model inspired by [11]. Our poetry generation approach can compose a poem in two steps.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Arabic Poem Generation Incorporating Deep Learning and Phonetic CNNsubword Embedding Models

Talafha¹,

Rekabdar²

2019

IJRC

View full text Add to dashboard Cite

Arabic poetry generation is a very challenging task since the linguistic structure of the Arabic language is considered a severe challenge for many researchers and developers in the Natural Language Processing (NLP) field. In this paper, we propose a poetry generation model with extended phonetic and semantic embeddings (Phonetic CNNsubword embeddings). We show that Phonetic CNNsubword embeddings have an effective contribution to the overall model performance compared to FastTextsubword embeddings. Our poetry generation model consists of a two-stage approach: (1.) generating the first verse which explicitly incorporates the theme related phrase, (2.) other verses generation with the proposed Hierarchy-Attention Sequence-to-Sequence model (HAS2S), which adequately capture word, phrase, and verse information between contexts. A comprehensive human evaluation confirms that the poems generated by our model outperform the base models in criteria such as Meaning, Coherence, Fluency, and Poeticness. Extensive quantitative experiments using Bi-Lingual Evaluation Understudy (BLEU) scores also demonstrate significant improvements over strong baselines.

show abstract

Subword analysis of small vocabulary and large vocabulary ASR for Punjabi language

Mittal

Singh²

2020

Int J Speech Technol

View full text Add to dashboard Cite

Subword-level Composition Functions for Learning Word Embeddings

Cited by 25 publications

References 34 publications

On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages

On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages

Arabic Poem Generation Incorporating Deep Learning and Phonetic CNNsubword Embedding Models

Subword analysis of small vocabulary and large vocabulary ASR for Punjabi language

Contact Info

Product

Resources

About