KILT: a Benchmark for Knowledge Intensive Language Tasks

Petroni, Filippo; Piktus, Aleksandra; Fan, Angela; Lewis, Patrick; Yazdani, Majid; Cao, Nicola De; Thorne, James; Jernite, Yacine; Karpukhin, Vladimir; Maillard, Jean; Plachouras, Vassilis; Rocktäschel, Tim; Riedel, Sebastian

doi:10.18653/v1/2021.naacl-main.200

Cited by 157 publications

(166 citation statements)

References 31 publications

Supporting

Mentioning

129

Contrasting

Order By: Relevance

“…Unlike using a single vector for each passage, DensePhrases represents each passage with multiple phrase vectors and the score of a passage can be computed by the maximum score of phrases within it. to outperform traditional sparse retrieval methods such as TF-IDF and BM25 in a range of knowledgeintensive NLP tasks (Petroni et al, 2021), including open-domain question answering (QA) (Chen et al, 2017), entity linking , and knowledge-grounded dialogue (Dinan et al, 2019).…”

Section: Has Been Shownmentioning

confidence: 99%

“…Following this positive finding, we further explore whether phrase retrieval can be extended to retrieval of coarser granularities, or other NLP tasks. Through fine-tuning of the query encoder with document-level supervision, we are able to obtain competitive performance on entity linking (Hoffart et al, 2011) and knowledge-grounded dialogue retrieval (Dinan et al, 2019) in the KILT benchmark (Petroni et al, 2021).…”

Section: Has Been Shownmentioning

confidence: 99%

“…Datasets We test DensePhrases trained with L doc on entity linking (Hoffart et al, 2011;Guo and Barbosa, 2018) and knowledge-grounded dialogue (Dinan et al, 2019) tasks in KILT (Petroni et al, 2021). Entity linking contains three datasets: AIDA CoNLL-YAGO (AY2) (Hoffart et al, 2011), WNED-WIKI (WnWi) (Guo and Barbosa, 2018), and WNED-CWEB (WnCw) (Guo and Barbosa, 2018).…”

Section: Improving Coarse-grained Retrievalmentioning

confidence: 99%

“…After training, DensePhrases returns the document that contains the top passage. For baseline retrieval methods, we report the performance of TF-IDF and DPR from Petroni et al (2021).…”

Section: Improving Coarse-grained Retrievalmentioning

confidence: 99%

“…We also include a multi-task version of DPR and DensePhrases, which uses the entire KILT training datasets. 11 While not our main focus of comparison, we also report the performance of other baselines from Petroni et al (2021), which uses generative models (e.g., RAG (Lewis et al, 2020)) or taskspecific models (e.g., BLINK , which has additional entity linking pre-training).…”

Section: Improving Coarse-grained Retrievalmentioning

confidence: 99%

See 4 more Smart Citations

Phrase Retrieval Learns Passage Retrieval, Too

Lee¹,

Wettig²,

Chen³

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Dense retrieval methods have shown great promise over sparse retrieval methods in a range of NLP problems. Among them, dense phrase retrieval-the most fine-grained retrieval unit-is appealing because phrases can be directly used as the output for question answering and slot filling tasks. 1 In this work, we follow the intuition that retrieving phrases naturally entails retrieving larger text blocks and study whether phrase retrieval can serve as the basis for coarse-level retrieval including passages and documents. We first observe that a dense phrase-retrieval system, without any retraining, already achieves better passage retrieval accuracy (+3-5% in top-5 accuracy) compared to passage retrievers, which also helps achieve superior end-to-end QA performance with fewer passages. Then, we provide an interpretation for why phrase-level supervision helps learn better fine-grained entailment compared to passage-level supervision, and also show that phrase retrieval can be improved to achieve competitive performance in document-retrieval tasks such as entity linking and knowledge-grounded dialogue. Finally, we demonstrate how phrase filtering and vector quantization can reduce the size of our index by 4-10x, making dense phrase retrieval a practical and versatile solution in multi-granularity retrieval. 2 * This work was done when JL worked as a visiting research scholar at Princeton University.1 Following previous work (Seo et al., 2018(Seo et al., , 2019, the term phrase denotes any contiguous text segment up to L words, which is not necessarily a linguistic phrase (see Section 2).2 Our code and models are available at https:// github.com/princeton-nlp/DensePhrases.

show abstract

Section: Has Been Shownmentioning

confidence: 99%

Section: Has Been Shownmentioning

confidence: 99%

Section: Improving Coarse-grained Retrievalmentioning

confidence: 99%

“…After training, DensePhrases returns the document that contains the top passage. For baseline retrieval methods, we report the performance of TF-IDF and DPR from Petroni et al (2021).…”

Section: Improving Coarse-grained Retrievalmentioning

confidence: 99%

Section: Improving Coarse-grained Retrievalmentioning

confidence: 99%

See 3 more Smart Citations

Phrase Retrieval Learns Passage Retrieval, Too

Lee¹,

Wettig²,

Chen³

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

Building trustworthy NeuroSymbolic AI Systems: Consistency, reliability, explainability, and safety

Gaur,

Sheth

2024

AI Magazine

View full text Add to dashboard Cite

Explainability and Safety engender trust. These require a model to exhibit consistency and reliability. To achieve these, it is necessary to use and analyze data and knowledge with statistical and symbolic AI methods relevant to the AI application––neither alone will do. Consequently, we argue and seek to demonstrate that the NeuroSymbolic AI approach is better suited for making AI a trusted AI system. We present the CREST framework that shows how Consistency, Reliability, user‐level Explainability, and Safety are built on NeuroSymbolic methods that use data and knowledge to support requirements for critical applications such as health and well‐being. This article focuses on Large Language Models (LLMs) as the chosen AI system within the CREST framework. LLMs have garnered substantial attention from researchers due to their versatility in handling a broad array of natural language processing (NLP) scenarios. As examples, ChatGPT and Google's MedPaLM have emerged as highly promising platforms for providing information in general and health‐related queries, respectively. Nevertheless, these models remain black boxes despite incorporating human feedback and instruction‐guided tuning. For instance, ChatGPT can generate unsafe responses despite instituting safety guardrails. CREST presents a plausible approach harnessing procedural and graph‐based knowledge within a NeuroSymbolic framework to shed light on the challenges associated with LLMs.

show abstract