Multilingual Autoregressive Entity Linking

Cao, Nicola De; Wu, Ledell; Popat, Kashyap; Artetxe, Mikel; Goyal, Naman; Plekhanov, Mikhail; Zettlemoyer, Luke; Cancedda, Nicola; Riedel, Sebastian; Petroni, Filippo

doi:10.1162/tacl_a_00460

Cited by 82 publications

(148 citation statements)

References 50 publications

(100 reference statements)

Supporting

Mentioning

147

Contrasting

Order By: Relevance

“…As shown in Table 2, models that were not trained on this specific task perform it very poorly. Entity linking is a task that is normally better performed by models which are explicitly designed for it (Cao et al, 2020). We nevertheless include it to showcase the ability of neural retrievers to adapt to it, and note how well the multi-task retriever performs on it in spite of its unusual nature.…”

Section: Universal Retrievalmentioning

confidence: 99%

See 1 more Smart Citation

Multi-task Retrieval for Knowledge-Intensive Tasks

Maillard¹,

Karpukhin²,

Petroni³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Retrieving relevant contexts from a large corpus is a crucial step for tasks such as opendomain question answering and fact checking. Although neural retrieval outperforms traditional methods like tf-idf and BM25, its performance degrades considerably when applied to out-of-domain data. Driven by the question of whether a neural retrieval model can be universal and perform robustly on a wide variety of problems, we propose a multi-task trained model. Our approach not only outperforms previous methods in the few-shot setting, but also rivals specialised neural retrievers, even when in-domain training data is abundant. With the help of our retriever, we improve existing models for downstream tasks and closely match or improve the state of the art on multiple benchmarks. * Equal Contribution. 1 While large pre-trained neural models have been shown to incorporate real-world knowledge in their parameters and thus may skip retrieval (Petroni et al., 2019), they still have limited capacity and suffer from a lack of explainability.

show abstract

Section: Universal Retrievalmentioning

confidence: 99%

“…Finally two entity linkers, GENRE (Cao et al, 2020) and BLINK , are worth mentioning. Being trained specifically for entity linking, these models will generally outperform retrieval-based approaches on that task.…”

Section: Related Workmentioning

confidence: 99%

Multi-task Retrieval for Knowledge-Intensive Tasks

Maillard¹,

Karpukhin²,

Petroni³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Considering the great scale and variable relations in Wikipedia, graph-based techniques generally don't apply, and big models are widely adopted in representing the text features. BLINK [132] encode contexts and entity descriptions via big models, while GENRE [133] builds an autoregressive model to directly output the unique name of chosen entities. These new models may also be viewed as "pre-trained entity linking models", as they are trained on the large-scale Wikipedia corpus, and can be easily transferred to specific datasets by fine-tuning on in-domain training data.…”

Section: Knowledge Graph Completion and Integrationmentioning

confidence: 99%

A Roadmap for Big Model

Yuan¹,

Zhao²,

Jiahong³

et al. 2022

Preprint

View full text Add to dashboard Cite

domains indexed by Google News. It contains 31 million documents with an average length of 793 BPE tokens. Like C4, it excludes examples with duplicate URLs. News dumps from December 2016 through March 2019 were used as training data, articles published in April 2019 from the April 2019 dump were used for evaluation. OpenWebText2(OWT2). OWT2 is an enhanced version of the original OpenWebTextCorpus, including content from multiple languages, document metadata, multiple dataset versions, and open source replication code, covering all Reddit submissions from 2005 up until April 2020. PubMed Central(PMC). PMC is a free full-text archive of biomedical and life sciences journal literature from the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM). The dataset is updated daily. In addition to full-text articles, they contain corrections, retractions, and expressions of concern, as well as file lists that include metadata for articles in each dataset.PMC obtained by open registration in Amazon Web Services (AWS) includes The PMC Open Access Subset and The Author Manuscript Dataset. The PMC Open Access Subset includes all articles and preprints in PMC with a machine-readable Creative Commons license that allows reuse. The Author Manuscript Dataset includes accepted author manuscripts collected under a funder policy in PMC and made available in machine-readable formats for text mining. ArXiv. ArXiv is a repository of 1.7 million articles, with relevant features such as article titles, authors, categories, abstracts, full text PDFs, and more. It provides open access to academic articles, covering many subdisciplines from vast branches of physics to computer science to everything in between, including math, statistics, electrical engineering, quantitative biology, and economics, which is helpful to the potential downstream applications of the research field. In addition, the writing language of LaTeX also contributes to the study of language models. Colossal Clean Crawled Corpus(C4). C4 is a colossal, cleaned version of Common Crawl's web crawl corpus. It is based on Common Crawl dataset and was used to train the T5 text-to-text Transformer models. The cleaned English version of C4 has 364,868,901 training examples and 364,608 validation examples, while the uncleaned English version has 1,063,805,324 training examples and 1,065,029 validation examples; the realnewslike version has 13,799,838 training examples and 13,863 validation examples, while the webtextlike version has 4,500,788 training examples and 4,493 validation examples. Wiki-40B. Wikipedia (Wiki-40B) is a clean-up text collection containing more than 40 Wikipedia language editions of pages corresponding to entities. The dataset is split into train/validation/test sets for each language. The training set has 2,926,536 examples, the validation set has 163,597 examples, and the test set has 162,274 examples. Wiki-40B is cleaned by a page filter to remove ambiguous, redirected, deleted, and non-physical pages. CLUECorpus2020. CLUECorpus2020 ...

show abstract

“…The inference of Seq2seq EL applies beam search (Sutskever et al, 2014) with targets constrained to the name set S by a prefix tree (constructed by name set S). Unlike mGENRE Cao et al (2021b) using provided candidates to decrease the size of the prefix tree, we use the whole name set S instead.…”

Section: Seq2seq Elmentioning

confidence: 99%

Generative Biomedical Entity Linking via Knowledge Base-Guided Pre-training and Synonyms-Aware Fine-tuning

Yuan¹,

Zheng²,

Yu³

2022

Preprint

View full text Add to dashboard Cite

Entities lie in the heart of biomedical natural language understanding, and the biomedical entity linking (EL) task remains challenging due to the fine-grained and diversiform concept names. Generative methods achieve remarkable performances in general domain EL with less memory usage while requiring expensive pre-training. Previous biomedical EL methods leverage synonyms from knowledge bases (KB) which is not trivial to inject into a generative method. In this work, we use a generative approach to model biomedical EL and propose to inject synonyms knowledge in it. We propose KBguided pre-training by constructing synthetic samples with synonyms and definitions from KB and require the model to recover concept names. We also propose synonyms-aware fine-tuning to select concept names for training, and propose decoder prompt and multisynonyms constrained prefix tree for inference. Our method achieves state-of-the-art results on several biomedical EL tasks without candidate selection which displays the effectiveness of proposed pre-training and fine-tuning strategies. The source code is available at Github.com/Yuanhy1997/GenBioEL.

show abstract

Multilingual Autoregressive Entity Linking

Cited by 82 publications

References 50 publications

Multi-task Retrieval for Knowledge-Intensive Tasks

Multi-task Retrieval for Knowledge-Intensive Tasks

A Roadmap for Big Model

Generative Biomedical Entity Linking via Knowledge Base-Guided Pre-training and Synonyms-Aware Fine-tuning

Contact Info

Product

Resources

About