Sanitizing Synthetic Training Data Generation for Question Answering over Knowledge Graphs

Abstract

Synthetic data generation is important to training and evaluating neural models for question answering over knowledge graphs. The quality of the data and the partitioning of the datasets into training, validation and test splits impact the performance of the models trained on this data. If the synthetic data generation depends on templates, as is the predominant approach for this task, there may be a leakage of information via a shared basis of templates across data splits if the partitioning is not performed hygienically. This paper investigates the extent of such information leakage across data splits, and the ability of trained models to generalize to test data when the leakage is controlled. We find that information leakage indeed occurs and that it affects performance. At the same time, the trained models do generalize to test data under the sanitized partitioning presented here. Importantly, these findings extend beyond the particular flavor of question answering task we studied and raise a series of difficult questions around template-based synthetic data generation that will necessitate additional research.

Would You Ask it that Way? Measuring and Improving Question Naturalness for Knowledge Graph Question Answering

“…We compare model performance on the original versus rewritten NL questions in our samples. Specifically, we use neural KGQA models trained on the DBNQA * [16] and GrailQA [11] datasets. 1 The question we seek to answer is how quality improvements on the input NL questions impact the answer prediction effectiveness of the models.…”

“…The extant datasets are taken as the basis to extract templates for both formal queries and NL questions, and those templates are then instantiated with different entity and predicate bindings. DBNQA* [16] partitions DBNQA [12] into training, validation, and test splits based on the underlying templates, avoiding leakage of information between training and test splits. The instances are identical to DBNQA, and so we use DBNQA* in our experiments.…”

“…We also choose the datasets so that all of the most common KGs are represented in the formal query bindings. Specifically, we consider the datasets DBNQA * [16], LC-QuAD v2.0 [9], and GrailQA [11]. We then randomly sample 25 NL questions from each of these datasets.…”

FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering

“…More broadly, in Knowledge-Graph Question Answering (KG-QA), work has exploited KG to generate synthetic data in unseen domains (Linjordet, 2020;Trivedi et al, 2017;Linjordet and Balog, 2020). Our work extends visually-grounded questions with valid common sense KG triplets.…”

How Train-Test Leakage Affects Zero-shot Retrieval

“…In practice, though, the datasets often still contain redundant data points. The redundancies inherent in text data, such as paraphrases, synonyms, etc., can be especially problematic, resulting in train-test leaks [19,24,29]. For instance, the training and test sets of the ELI5 dataset [13] for question answering were created using TF-IDF as a heuristic to eliminate redundancies between them.…”