Benchmarking Ontologies: Bigger or Better?

Abstract

A scientific ontology is a formal representation of knowledge within a domain, typically including central concepts, their properties, and relations. With the rise of computers and high-throughput data collection, ontologies have become essential to data mining and sharing across communities in the biomedical sciences. Powerful approaches exist for testing the internal consistency of an ontology, but not for assessing the fidelity of its domain representation. We introduce a family of metrics that describe the breadth and depth with which an ontology represents its knowledge domain. We then test these metrics using (1) four of the most common medical ontologies with respect to a corpus of medical documents and (2) seven of the most popular English thesauri with respect to three corpora that sample language from medicine, news, and novels. Here we show that our approach captures the quality of ontological representation and guides efforts to narrow the breach between ontology and collective discourse within a domain. Our results also demonstrate key features of medical ontologies, English thesauri, and discourse from different domains. Medical ontologies have a small intersection, as do English thesauri. Moreover, dialects characteristic of distinct domains vary strikingly as many of the same words are used quite differently in medicine, news, and novels. As ontologies are intended to mirror the state of knowledge, our methods to tighten the fit between ontology and domain will increase their relevance for new areas of biomedical science and improve the accuracy and power of inferences computed across them.

War of Ontology Worlds: Mathematics, Computer Code, or Esperanto?

“…These challenges suggest a new virtue, most consistent with the Esperanto perspective: representativeness [20]. Insofar as ontologies are employed not only to index biomedical knowledge, but to discover it, they must maintain inconsistent biomedical claims, just as research scientists attempt to do.…”

Similarity algorithm for evaluating the coverage of domain ontology for semantic Web services

“…Research done by [25] hybrid several ontology evaluation approach including synonym types of similarity measures that focused on biomedical domain ontology. There are large numbers of domain ontology in Web, for example Hotel related ontology [26], service ontology [27] and Quran ontology [30].…”

Measuring the Evolution of Ontology Complexity: The Gene Ontology Case Study

“…Shchekotykhin et al proposed an entropy-based approach for localizing faults when debugging ontologies [8]. Yao et al formally defined metrics of an ontology's fit with respect to published knowledge in the form of other ontologies and of scientific articles [9]. Hoehndorf et al propose a method to evaluate biomedical ontologies for a particular problem by quantifying the success of using the ontology for this problem [10].…”

“…Yao et al provide a review of ontology evaluation and identified four categories: (1) measures of an ontology's internal consistency, (2) usability and task-based performance, (3) comparison with other ontologies and (4) match to reality [9].…”

“…We propose an approach based on ontology complexity. Compared to Yao et al's four categories of ontology evaluation [9], it offers a complementary view but is different from ontology's internal consistency.…”