Compositional Generalization in Multilingual Semantic Parsing over Wikidata

Cui, Ruixiang; Aralikatte, Rahul; Lent, Heather; Hershcovich, Daniel

doi:10.48550/arxiv.2108.03509

Cited by 4 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other KGQA datasets are Free917 (Cai and Yates, 2013), WebQuestions (Berant et al, 2013), Com-plexQuestions (Bao et al, 2016), SimplesQuestions (Bordes et al, 2015), GraphQuestions (Su et al, 2016), WebQuestionsSP (Yih et al, 2016), 30MFactoidQA (Serban et al, 2016), ComplexWebQuestions (Talmor and Berant, 2018), PathQuestion (Zhou et al, 2018), MetaQA (Zhang et al, 2018), TempQuestions (Jia et al, 2018), TimeQuestions (Jia et al, 2021), Cron-Questions (Saxena et al, 2021), FreebaseQA (Jiang et al, 2019), Compositional Freebase Questions (CFQ) (Keysers et al, 2019), Compositional Wikidata Questions (CWQ) (Cui et al, 2021), RuBQ (Korablinov and Braslavski, 2020;Rybin et al, 2021), GrailQA (Gu et al, 2021), Event-QA (Souza Costa et al, 2020), Sim-pleDBpediaQA (Azmy et al, 2018), CLC-QuAD (Zou et al, 2021), KQA Pro (Shi et al, 2020), SimpleQues-tionsWikidata (Diefenbach et al, 2017), DBNQA (Yin et al, 2019), etc. These datasets do not fulfill our current criteria and thus are not part of the initial version of the KGQA leaderboard.…”

Section: Kgqa Datasetsmentioning

confidence: 99%

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

Perevalov¹,

Yan²,

Kovriguina³

et al. 2022

Preprint

View full text Add to dashboard Cite

Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community -https://kgqa.github.io/leaderboard/. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.

show abstract

Section: Kgqa Datasetsmentioning

confidence: 99%

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

Perevalov¹,

Yan²,

Kovriguina³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Today, the research in the field of KGQA is strongly dependent on data, and it suffers from the lack of multilingual benchmarks [6], [7]. To the best of our knowledge, only three KGQA benchmarks exist that tackle multiple languages: QALD [3], RuBQ [4], and CWQ [5].…”

Section: Multilingual Kgqa Benchmarksmentioning

confidence: 99%

“…One of them is the lack of available benchmarks. To the best of our knowledge, there are only 3 mKGQA datasets in the research community (QALD-9 [3], RuBQ 2.0 [4], and CWQ [5]) and all of them are not fulfilling completely the practical needs of researchers and developers (see Section II). Hence, even if one develops an mKGQA system, there are not so many opportunities for full-fledged evaluation.…”

Section: Introductionmentioning

confidence: 99%

QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers

Perevalov¹,

Diefenbach²,

Usbeck³

et al. 2022

Preprint

View full text Add to dashboard Cite

The ability to have the same experience for different user groups (i.e., accessibility) is one of the most important characteristics of Web-based systems. The same is true for Knowledge Graph Question Answering (KGQA) systems that provide the access to Semantic Web data via natural language interface. While following our research agenda on the multilingual aspect of accessibility of KGQA systems, we identified several ongoing challenges. One of them is the lack of multilingual KGQA benchmarks. In this work, we extend one of the most popular KGQA benchmarks -QALD-9 by introducing highquality questions' translations to 8 languages provided by native speakers, and transferring the SPARQL queries of QALD-9 from DBpedia to Wikidata, s.t., the usability and relevance of the dataset is strongly increased. Five of the languages -Armenian, Ukrainian, Lithuanian, Bashkir and Belarusian -to our best knowledge were never considered in KGQA research community before. The latter two of the languages are considered as "endangered" by UNESCO. We call the extended dataset QALD-9-plus and made it available online 1 .

show abstract

“…MCWQ (Cui et al, 2022), a multilingual variant of CFQ, is the first adaptation into multiple languages of a compositional generalisation benchmark. It was created with the use of neural machine translation.…”

Section: Introductionmentioning

confidence: 99%

mSCAN: A Dataset for Multilingual Compositional Generalisation Evaluation

Reymond,

Steinert-Threlkeld

2023

Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP

View full text Add to dashboard Cite

Language models achieve remarkable results on a variety of tasks, yet still struggle with compositional generalisation benchmarks. The majority of these benchmarks evaluate performance in English only, leaving us with the question of whether these results generalise to other languages. As an initial step to answering this question, we introduce mSCAN, a multilingual adaptation of the SCAN dataset. It was produced by a rule-based translation, developed in cooperation with native speakers. We then showcase this novel dataset on some in-context learning experiments, with the multilingual large language model BLOOM as well as gpt3.5-turbo.

show abstract

Compositional Generalization in Multilingual Semantic Parsing over Wikidata

Cited by 4 publications

References 0 publications

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers

mSCAN: A Dataset for Multilingual Compositional Generalisation Evaluation

Contact Info

Product

Resources

About