Database reasoning over text

Abstract

Neural models have shown impressive performance gains in answering queries from natural language text. However, existing works are unable to support database queries, such as "List/Count all female athletes who were born in 20th century", which require reasoning over sets of relevant facts with operations such as join, filtering and aggregation. We show that while state-of-the-art transformer models perform very well for small databases, they exhibit limitations in processing noisy data, numerical operations, and queries that aggregate facts. We propose a modular architecture to answer these database-style queries over multiple spans from text and aggregating these at scale. We evaluate the architecture using WIKINLDB, 1 a novel dataset for exploring such queries. Our architecture scales to databases containing thousands of facts whereas contemporary models are limited by how many facts can be encoded. In direct comparison on small databases, our approach increases overall answer accuracy from 85% to 90%. On larger databases, our approach retains its accuracy whereas transformer baselines could not encode the context.

The NLP Task Effectiveness of Long-Range Transformers

“…These models have been evaluated primarily via perplexity and non-NLP benchmarks (Tay et al, 2020b). These metrics may not be ideal (Sun et al, 2021) and may not reflect performance on complex NLP tasks (Arutiunian et al, 2020;Thorne et al, 2021). We argue these metrics have not been sufficient for the development of efficient Transformers and their practical application on long texts, and that existing benchmarks are insufficient guides for architecture selection.…”

Multimodal Neural Databases

“…Answering Database Queries There has been substantial effort put into converting queries expressed in natural language into SQL queries for databases with known structure [1,16,38], and there have also been advancements in adapting this approach for semistructured data and knowledge bases [4,20]. Recently, Thorne et al [28,29] proposed NeuralDB as a way to perform database queries over a collection of textual documents without the need to translate data or queries into a predefined database schema but using parallel neural techniques instead. Their approach is very effective but it: (i) requires preprocessing and analysis for the aggregation operator; (ii) is limited to simple queries and (iii) is capable of handling data just in textual format.…”

“…In this perspective paper, we propose to study, design, and build MMNDBs by combining the capabilities of large multimodal models, multi-media information retrieval, and database query processing, as shown in Figure 1. We have been inspired by the work on neural databases [26,28,29] that have garnered interest in the NLP, database, and IR communities. However, we differentiate from that work as we position ourselves as an evolution of the field of MMIR by means of modern and, more recently proposed, multimodal AI technologies.…”

Compositional Generalization in Multilingual Semantic Parsing over Wikidata

“…While the dataset is bilingual, it uses crowdsourced questions and is not designed for compositionality analysis. Recently, Thorne et al (2021) proposed WIKINLDB, a Wikidata-based English KBQA dataset, focusing on scalability rather than compositionality. Other related datasets include QALM (Kaffee et al, 2019), a dataset for multilingual question answering over a set of different popular knowledge graphs, intended to help determine the multilinguality of those knowledge graphs.…”