Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

Wang, Ling; Luís, Tiago; Marujo, Luís; Astudillo, Ramón Fernandez; Amir, Sílvio; Dyer, Chris; Black, Alan W.; Trancoso, Isabel

doi:10.48550/arxiv.1508.02096

Cited by 90 publications

(53 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The single BTS model improves on average over the CRF models trained using the same data, though clearly there is some benefit in using external resources. Note that BTS is particularly strong in Finnish, surpassing even CRF+ by nearly 1.5% (absolute), probably because the byte representation generalizes better to agglutinative languages than word-based models, a finding validated by Ling et al (2015). In addition, the baseline CRF models, including the (compressed) cluster tables, require about 50 MB per language, while BTS is under 10 MB.…”

Section: Part-of-speech Taggingmentioning

confidence: 93%

“…Recent work has shown that modeling the sequence of characters in each token with an LSTM can more effectively handle rare and unknown words than independent word embeddings (Ling et al, 2015;Ballesteros et al, 2015). Similarly, language modeling, especially for morphologically complex languages, benefits from a Convolutional Neural Network (CNN) over characters to generate word embeddings (Kim et al, 2015).…”

Section: Related Workmentioning

confidence: 99%

“…Our part-of-speech tagging experiments use Version 1.1 of the Universal Dependency data 5 , a collection of treebanks across many languages annotated with a universal tagset (Petrov et al, 2011). The most relevant recent work (Ling et al, 2015) uses different datasets, with different finer-grained tagsets in each language. Because we are primary interested in multilingual models that can share languageindependent parameters, the universal tagset is important, and thus our results are not immediately comparable.…”

Section: Part-of-speech Taggingmentioning

confidence: 99%

See 2 more Smart Citations

Multilingual Language Processing From Bytes

Gillick

Brunk

Vinyals

et al. 2016

Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

179

120

View full text Add to dashboard Cite

We describe an LSTM-based model which we call Byte-to-Span (BTS) that reads text as bytes and outputs span annotations of the form [start, length, label] where start positions, lengths, and labels are separate entries in our vocabulary. Because we operate directly on unicode bytes rather than languagespecific words or characters, we can analyze text in many languages with a single model. Due to the small vocabulary size, these multilingual models are very compact, but produce results similar to or better than the state-ofthe-art in Part-of-Speech tagging and Named Entity Recognition that use only the provided training datasets (no external data sources). Our models are learning "from scratch" in that they do not rely on any elements of the standard pipeline in Natural Language Processing (including tokenization), and thus can run in standalone fashion on raw text.

show abstract

Section: Part-of-speech Taggingmentioning

confidence: 93%

Section: Related Workmentioning

confidence: 99%

Section: Part-of-speech Taggingmentioning

confidence: 99%

See 1 more Smart Citation

Multilingual Language Processing From Bytes

Gillick

Brunk

Vinyals

et al. 2016

Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

179

120

View full text Add to dashboard Cite

show abstract

“…Jozefowicz et al (2015) address this limitation in their "Big LM" architecture by representing words as points in a lower-dimensional continuous space using a convolutional neural network (CNN). These models can represent words as embeddings (real-valued vectors) that capture perceptual similarity between words on the basis of shared structure, such as word lemmas or part of speech markings like -ing (Ling et al, 2015). The LSTM is then used to make predictions in this lower dimensional space, reducing the number of computations in the softmax function from the size of the vocabulary to the dimensionality of the CNN-derived embeddings.…”

Section: Recurrent Neural Network Language Model (Rnn Lm)mentioning

confidence: 99%

Evaluating Models of Robust Word Recognition with Serial Reproduction

Meylan,

Nair,

Griffiths

2021

Preprint

View full text Add to dashboard Cite

Spoken communication occurs in a "noisy channel" characterized by high levels of environmental noise, variability within and between speakers, and lexical and syntactic ambiguity. Given these properties of the received linguistic input, robust spoken word recognition-and language processing more generally-relies heavily on listeners' prior knowledge to evaluate whether candidate interpretations of that input are more or less likely. Here we compare several broad-coverage probabilistic generative language models in their ability to capture human linguistic expectations. Serial reproduction, an experimental paradigm where spoken utterances are reproduced by successive participants similar to the children's game of "Telephone," is used to elicit a sample that reflects the linguistic expectations of English-speaking adults. When we evaluate a suite of probabilistic generative language models against the yielded chains of utterances, we find that those models that make use of abstract representations of preceding linguistic context (i.e., phrase structure) best predict the changes made by people in the course of serial reproduction. A logistic regression model predicting which words in an utterance are most likely to be lost or changed in the course of spoken transmission corroborates this result.We interpret these findings in light of research highlighting the interaction of memory-based constraints and representations in language processing.

show abstract

“…Launching a novel drug into the market would take more than ten years on average, with a substantial investment of billions of USD [ 2 ]. Meanwhile, deep learning shows great success in other fields, such as natural-language processing [ 3 , 4 , 5 , 6 , 7 , 8 , 9 ] and pattern recognition [ 10 , 11 , 12 , 13 ], as well as the improvement of computing power and dataset availability. Its potential in promoting efficiency and success rate of drug development, in particular the prediction of molecular properties, has been widely investigated for years [ 14 , 15 , 16 , 17 , 18 ].…”

Section: Introductionmentioning

confidence: 99%

Accurate Physical Property Predictions via Deep Learning

et al. 2022

View full text Add to dashboard Cite

Neural networks and deep learning have been successfully applied to tackle problems in drug discovery with increasing accuracy over time. There are still many challenges and opportunities to improve molecular property predictions with satisfactory accuracy even further. Here, we proposed a deep-learning architecture model, namely Bidirectional long short-term memory with Channel and Spatial Attention network (BCSA), of which the training process is fully data-driven and end to end. It is based on data augmentation and SMILES tokenization technology without relying on auxiliary knowledge, such as complex spatial structure. In addition, our model takes the advantages of the long- and short-term memory network (LSTM) in sequence processing. The embedded channel and spatial attention modules in turn specifically identify the prime factors in the SMILES sequence for predicting properties. The model was further improved by Bayesian optimization. In this work, we demonstrate that the trained BSCA model is capable of predicting aqueous solubility. Furthermore, our proposed method shows noticeable superiorities and competitiveness in predicting oil–water partition coefficient, when compared with state-of-the-art graphs models, including graph convoluted network (GCN), message-passing neural network (MPNN), and AttentiveFP.

show abstract

Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

Cited by 90 publications

References 12 publications

Multilingual Language Processing From Bytes

Multilingual Language Processing From Bytes

Evaluating Models of Robust Word Recognition with Serial Reproduction

Accurate Physical Property Predictions via Deep Learning

Contact Info

Product

Resources

About