RuBQ 2.0: An Innovated Russian Question Answering Dataset

Rybin, Ivan; Korablinov, Vladislav; Efimov, Pavel; Braslavski, Pavel

doi:10.1007/978-3-030-77385-4_32

Cited by 12 publications

(7 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other KGQA datasets are Free917 (Cai and Yates, 2013), WebQuestions (Berant et al, 2013), Com-plexQuestions (Bao et al, 2016), SimplesQuestions (Bordes et al, 2015), GraphQuestions (Su et al, 2016), WebQuestionsSP (Yih et al, 2016), 30MFactoidQA (Serban et al, 2016), ComplexWebQuestions (Talmor and Berant, 2018), PathQuestion (Zhou et al, 2018), MetaQA (Zhang et al, 2018), TempQuestions (Jia et al, 2018), TimeQuestions (Jia et al, 2021), Cron-Questions (Saxena et al, 2021), FreebaseQA (Jiang et al, 2019), Compositional Freebase Questions (CFQ) (Keysers et al, 2019), Compositional Wikidata Questions (CWQ) (Cui et al, 2021), RuBQ (Korablinov and Braslavski, 2020;Rybin et al, 2021), GrailQA (Gu et al, 2021), Event-QA (Souza Costa et al, 2020), Sim-pleDBpediaQA (Azmy et al, 2018), CLC-QuAD (Zou et al, 2021), KQA Pro (Shi et al, 2020), SimpleQues-tionsWikidata (Diefenbach et al, 2017), DBNQA (Yin et al, 2019), etc. These datasets do not fulfill our current criteria and thus are not part of the initial version of the KGQA leaderboard.…”

Section: Kgqa Datasetsmentioning

confidence: 99%

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

Perevalov¹,

Yan²,

Kovriguina³

et al. 2022

Preprint

View full text Add to dashboard Cite

Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community -https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.

show abstract

Section: Kgqa Datasetsmentioning

confidence: 99%

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

Perevalov¹,

Yan²,

Kovriguina³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, languages besides English and Chinese are not covered and the work does not provide a deeper analysis of the issues with the SPARQL query generation process faced when working with Wikidata. The RuBQ benchmark series [17,26] which was initially based on questions from Russian quizzes (totaling 2,910 questions) has also been translated to English via machine translation. The SPARQL queries over Wikidata were generated automatically and manually validated by the authors.…”

Section: Multilinguality In Kgqamentioning

confidence: 99%

QALD-10 – The 10th challenge on question answering over linked data

Usbeck,

Yan,

Perevalov

et al. 2023

View full text Add to dashboard Cite

Knowledge Graph Question Answering (KGQA) has gained attention from both industry and academia over the past decade. Researchers proposed a substantial amount of benchmarking datasets with different properties, pushing the development in this field forward. Many of these benchmarks depend on Freebase, DBpedia, or Wikidata. However, KGQA benchmarks that depend on Freebase and DBpedia are gradually less studied and used, because Freebase is defunct and DBpedia lacks the structural validity of Wikidata. Therefore, research is gravitating toward Wikidata-based benchmarks. That is, new KGQA benchmarks are created on the basis of Wikidata and existing ones are migrated. We present a new, multilingual, complex KGQA benchmarking dataset as the 10th part of the Question Answering over Linked Data (QALD) benchmark series. This corpus formerly depended on DBpedia. Since QALD serves as a base for many machine-generated benchmarks, we increased the size and adjusted the benchmark to Wikidata and its ranking mechanism of properties. These measures foster novel KGQA developments by more demanding benchmarks. Creating a benchmark from scratch or migrating it from DBpedia to Wikidata is non-trivial due to the complexity of the Wikidata knowledge graph, mapping issues between different languages, and the ranking mechanism of properties using qualifiers. We present our creation strategy and the challenges we faced that will assist other researchers in their future work. Our case study, in the form of a conference challenge, is accompanied by an in-depth analysis of the created benchmark.

show abstract

“…Today, the research in the field of KGQA is strongly dependent on data, and it suffers from the lack of multilingual benchmarks [6], [7]. To the best of our knowledge, only three KGQA benchmarks exist that tackle multiple languages: QALD [3], RuBQ [4], and CWQ [5].…”

Section: Multilingual Kgqa Benchmarksmentioning

confidence: 99%

“…One of them is the lack of available benchmarks. To the best of our knowledge, there are only 3 mKGQA datasets in the research community (QALD-9 [3], RuBQ 2.0 [4], and CWQ [5]) and all of them are not fulfilling completely the practical needs of researchers and developers (see Section II). Hence, even if one develops an mKGQA system, there are not so many opportunities for full-fledged evaluation.…”

Section: Introductionmentioning

confidence: 99%

“…To create the extension, a number of crowd workers were asked to translate the questions into their mother tongue manually (and validate translations by others). As a result, 17 volunteer participants and 290 crowd workers from Amazon Mechanical Turk 4 and Yandex Toloka 5 translated questions into 8 languages, such as: German, Russian, French, Armenian, Belarusian, Lithuanian, Bashkir, and Ukrainian, the latter five of which, to our knowledge, have never been considered in the KGQA context. In addition, the SPARQL queries in the dataset were transformed from DBpedia to Wikidata in order to increase the usability of the dataset.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers

Perevalov¹,

Diefenbach²,

Usbeck³

et al. 2022

Preprint

View full text Add to dashboard Cite

The ability to have the same experience for different user groups (i.e., accessibility) is one of the most important characteristics of Web-based systems. The same is true for Knowledge Graph Question Answering (KGQA) systems that provide the access to Semantic Web data via natural language interface. While following our research agenda on the multilingual aspect of accessibility of KGQA systems, we identified several ongoing challenges. One of them is the lack of multilingual KGQA benchmarks. In this work, we extend one of the most popular KGQA benchmarks -QALD-9 by introducing highquality questions' translations to 8 languages provided by native speakers, and transferring the SPARQL queries of QALD-9 from DBpedia to Wikidata, s.t., the usability and relevance of the dataset is strongly increased. Five of the languages -Armenian, Ukrainian, Lithuanian, Bashkir and Belarusian -to our best knowledge were never considered in KGQA research community before. The latter two of the languages are considered as "endangered" by UNESCO. We call the extended dataset QALD-9-plus and made it available online 1 .

show abstract

RuBQ 2.0: An Innovated Russian Question Answering Dataset

Cited by 12 publications

References 20 publications

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

QALD-10 – The 10th challenge on question answering over linked data

QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers

Contact Info

Product

Resources

About