QALD-10 – The 10th challenge on question answering over linked data

Abstract

Knowledge Graph Question Answering (KGQA) has gained attention from both industry and academia over the past decade. Researchers proposed a substantial amount of benchmarking datasets with different properties, pushing the development in this field forward. Many of these benchmarks depend on Freebase, DBpedia, or Wikidata. However, KGQA benchmarks that depend on Freebase and DBpedia are gradually less studied and used, because Freebase is defunct and DBpedia lacks the structural validity of Wikidata. Therefore, research is gravitating toward Wikidata-based benchmarks. That is, new KGQA benchmarks are created on the basis of Wikidata and existing ones are migrated. We present a new, multilingual, complex KGQA benchmarking dataset as the 10th part of the Question Answering over Linked Data (QALD) benchmark series. This corpus formerly depended on DBpedia. Since QALD serves as a base for many machine-generated benchmarks, we increased the size and adjusted the benchmark to Wikidata and its ranking mechanism of properties. These measures foster novel KGQA developments by more demanding benchmarks. Creating a benchmark from scratch or migrating it from DBpedia to Wikidata is non-trivial due to the complexity of the Wikidata knowledge graph, mapping issues between different languages, and the ranking mechanism of properties using qualifiers. We present our creation strategy and the challenges we faced that will assist other researchers in their future work. Our case study, in the form of a conference challenge, is accompanied by an in-depth analysis of the created benchmark.

Enhancing SPARQL Query Generation for Knowledge Base Question Answering Systems by Learning to Correct Triplets

“…In this section, we present the experimental evaluation conducted on three extensively utilized KBQA datasets, namely LC-QuAD 2.0 [18], QALD-9 plus [19], and QALD-10 [20], to assess the viability and efficacy of our proposed TSET model. We begin by introducing the experimental settings, encompassing the datasets used, the evaluation metric employed, and the implementation details.…”

“…Experimental results show that our model can significantly improve the quality of SPARQL query generation. On three well-known KBQA datasets, LC-QuAD 2.0 [18], QALD-9 plus [19], and QALD-10 [20], TSET surpasses all previous methods in answer F1 score and Query Match (QM) accuracy, achieving new state-of-the-art performance. We also do a comprehensive set of ablation studies to demonstrate the effectiveness of our proposed Triple Structure Correction (TSC) objective and the semantic transformation approach.…”

“…In this work, we utilize three widely used datasets, all based on Wikidata: LC-QuAD 2.0 [18], QALD-9 plus [19], and QALD-10 [20].…”

FLMatchQA: a recursive neural network-based question answering with customized federated learning model

“…The author proposes adding a proximal term to local objectives to address non-IID data and heterogeneous updates. They use server and client control to estimate update directions, mimicking centralized methods and normalizing and scaling client updates before updating the global model ( Usbeck et al, 2023 ). Azad & Deepak (2019) surveyed Natural Language Interfaces for databases (NLIDB) for QA systems, not KGQA systems, using a set of 10 questions to evaluate 24 QA systems and compare them to other SQL query conversion systems.…”