Are Linked Datasets fit for Open-domain Question Answering? A Quality Assessment

Abstract

The current decade is a witness to an enormous explosion of data being published on the Web as Linked Data to maximise its reusability. Answering questions that users speak or write in natural language is an increasingly popular application scenario for Web Data, especially when the domain of the questions is not limited to a domain where dedicated curated datasets exist, like in medicine. The increasing use of Web Data in this and other settings has highlighted the importance of assessing its quality. While quite some work has been done with regard to assessing the quality of Linked Data, only few efforts have been dedicated to quality assessment of linked data from the question answering (QA) perspective. From the linked data quality metrics that have so far been well documented in the literature, we have identified those that are most relevant for QA. We apply these quality metrics, implemented in the Luzzu framework, to subsets of two datasets of crucial importance to open domain QA-DBpedia and Wikidata-and thus present the first assessment of the quality of these datasets for QA. From these datasets, we assess slices covering the specific domains of restaurants, politicians, films and soccer players. The results of our experiments suggest that for most of these domains, the quality of Wikidata with regard to the majority of relevant metrics is higher than that of DBpedia.

A systematic literature review on Wikidata

“…For Ringler and Paulheim (2017), although “DBpedia, YAGO, or Wikidata, are often considered similar in nature and coverage, there are, in fact, quite a few differences.” In their work, they quantified those differences and identified the overlapping and complementary parts of these three public knowledge graphs, finding them “hardly interchangeable,” each with advantages and issues that depended on the desired application or domain. Abián et al (2018) also compared Wikidata with DBpedia in regard to “the most relevant data quality dimensions” and highlighted how Wikidata has “an open centralised nature” and its multi-lingual capacity, while DBpedia is “more popular in the Semantic Web and the Linked Open Data communities.” On the other hand, Thakkar et al (2016) ran a quality assessment of linked data in DBpedia and Wikidata from the perspective of question answering, a popular application scenario for knowledge databases, and found “the quality of Wikidata with regard to the majority of relevant metrics […] higher than that of DBpedia.”…”

Question Answering on Linked Data

“…Data quality dimensions in all of these categories can be relevant in question answering scenarios. In a preliminary study [51], we evaluated few selected metrics mentioned above on two popular datasets of linked data namely, Wikidata and DBpedia 13 . We evaluated the subsections extracted from these datasets on categories such as "politicians", "Movies", "Restaurants" and "Soccer players".…”

“…The slices have been made available to the common public and can be found: https://goo.gl/Kn6Fom (DBpedia slices), https://goo.gl/5aTkLp (Wikidata slices). The detailed information regarding the data slice statistics can be found from the work [51] which is selected from a detailed report made public at https://goo.gl/ignzzI. For this evaluation we have obtained four slices of both DBpedia and Wikidata, namely Restaurants, Politicians, Films and Soccer players.…”

“…Regarding implementation, in our recent study [51], we evaluated the results on DBpedia and Wikidata slices using the metrics that the Luzzu linked data quality assessment framework already provides. We look forward to extending Luzzu to use in a question answering setting, that further existing implementations of metrics in Luzzu can be specifically adapted to make them suitable for quality assessment related to question answering, and that, finally, Luzzu's flexible extensibility even enables us to implement new metrics that may be required.…”

Evaluation of the Linked Open Data Quality Based on a Fuzzy Logic Model

“…Due to this large number of people, which can upload data to the Web, this model has grown exponentially, from 12 datasets in 2007, to approximately 300 in September 2011, and 9,960 datasets in 2016 [3], taking into account that this number is accumulated from the data catalogs: data.gov, publicdata.eu and datahub.io [4]; three of the main collections of datasets available to the public. This growth in Linked Open Data requires methods or tools that can handle this data, to make it "secure, stable, fast and accurate; in other words, have a better quality," [5]. However, quality is a very complex concept; defining what is good quality or not is very subjective and cannot be defined in one way or by a simple judgment [2,6].…”