Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

Alon, Uri; Xu, Frank F.; He, Jun; Sengupta, Sudipta; Roth, Dan; Neubig, Graham

doi:10.48550/arxiv.2201.12431

Cited by 2 publications

(3 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, Retrieval-based language models (R-LMs) [1] have risen to improve over language models in various tasks such as unconditional language modeling [14,25], machine translation [10,24], text classification [27] and question answering [23]. R-LMs is a non-parametric method that uses training examples to augment the language model predictions at test time.…”

Section: Retrieval-based Language Modelsmentioning

confidence: 99%

Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning

Chen,

Li,

Zhang

et al. 2022

Preprint

View full text Add to dashboard Cite

Pre-trained language models have contributed significantly to relation extraction by demonstrating remarkable few-shot learning abilities. However, prompt tuning methods for relation extraction may still fail to generalize to those rare or hard patterns. Note that the previous parametric learning paradigm can be viewed as memorization regarding training data as a book and inference as the close-book test. Those long-tailed or hard patterns can hardly be memorized in parameters given few-shot instances. To this end, we regard RE as an open-book examination and propose a new semiparametric paradigm of retrieval-enhanced prompt tuning for relation extraction. We construct an open-book datastore for retrieval regarding prompt-based instance representations and corresponding relation labels as memorized key-value pairs. During inference, the model can infer relations by linearly interpolating the base output of PLM with the non-parametric nearest neighbor distribution over the datastore. In this way, our model not only infers relation through knowledge stored in the weights during training but also assists decision-making by unwinding and querying examples in the open-book datastore. Extensive experiments on benchmark datasets show that our method can achieve state-of-the-art in both standard supervised and few-shot settings 1 .

show abstract

Section: Retrieval-based Language Modelsmentioning

confidence: 99%

Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning

Chen,

Li,

Zhang

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, retrieval-augmented LMs have shown a series of impressive results (Grave et al, 2017;Guu et al, 2018;He et al, 2020;Khandelwal et al, 2020b;Borgeaud et al, 2022;Alon et al, 2022). Retrieval-augmented LMs compute next token distributions based not only on the immediately preceding context c t and the model parameters, but also on an external datastore, from which examples are retrieved and incorporated into the base LM's prediction.…”

Section: Introductionmentioning

confidence: 99%

“…Finally, one significant drawback to the current kNN-LM is the inefficiency of kNN search performed at each step (He et al, 2021;Borgeaud et al, 2022;Alon et al, 2022;Wang et al, 2022). Because of the similarity between kNN-LM and the parametric LM's last layers and the many design choices, we also demonstrate that we are able to make kNN-LM more efficient by substituting the kNN search with another matrix operation that can fit in accelerator memory while maintaining more than half the perplexity improvement, or more than 6.5% relative improvement compared to the base LM.…”

Section: Introductionmentioning

confidence: 99%

Why do Nearest Neighbor Language Models Work?

Xu¹,

Alon²,

Neubig³

2023

Preprint

View full text Add to dashboard Cite

Language models (LMs) compute the probability of a text by sequentially computing a representation of an already-seen context and using this representation to predict the next word. Currently, most LMs calculate these representations through a neural network consuming the immediate previous context. However recently, retrieval-augmented LMs have shown to improve over standard neural LMs, by accessing information retrieved from a large datastore, in addition to their standard, parametric, next-word prediction. In this paper, we set out to understand why retrieval-augmented language models, and specifically why k-nearest neighbor language models (kNN-LMs) perform better than standard parametric LMs, even when the k-nearest neighbor component retrieves examples from the same training set that the LM was originally trained on. To this end, we perform a careful analysis of the various dimensions over which kNN-LM diverges from standard LMs, and investigate these dimensions one by one. Empirically, we identify three main reasons why kNN-LM performs better than standard LMs: using a different input representation for predicting the next tokens, approximate kNN search, and the importance of softmax temperature for the kNN distribution. Further, we incorporate these insights into the model architecture or the training procedure of the standard parametric LM, improving its results without the need for an explicit retrieval component. The code is available at https://github.com/frankxu2004/knnlm-why.

show abstract

Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

Cited by 2 publications

References 20 publications

Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning

Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning

Why do Nearest Neighbor Language Models Work?

Contact Info

Product

Resources

About