Evaluating Ontologies: Towards a Cognitive Measure of Quality

Abstract

a b s t r a c tOntologies are formal specifications of shared conceptualizations of a domain. Important applications of ontologies include distributed knowledge-based systems, such as the semantic web, and the evaluation of modelling languages, e.g. for business process or conceptual modelling. These applications require formal ontologies of good quality. The quality of a formal ontology requires both a good conceptualization of a domain and a good specification of the conceptualization. In this paper, we focus on the latter aspect, and present a method to test how well a specification of a formal ontology corresponds to a conceptualization of a domain held by ontology users. Our experimental method is based on principles of cognitive psychology. We present two experiments to demonstrate our method using upper-level ontologies.

Using ontologies to model human navigation behavior in information networks: A study based on Wikipedia

“…Research in psychology suggests that humans store concepts in their minds hierarchically [7]. In our method, we model different groups of users by using different ontologies as background knowledge.…”

Developing an Ontology Evaluation Methodology: Cognitive Measure of Quality

“…Fang made an initial attempt at developing an ontology evaluation methodology that tests how well a specification of an ontology corresponds to the ontology users' conceptualization of a domain, and whether the ontological structures reflect the perceived reality of the world [28,27,26]. A sentence verification technique adopted from cognitive psychology (described in Section 3.2.2) was applied to compare two upper level ontologies (the most general and abstract form of ontologies), SUMO and BWW [28,27,26]; and in Evermann and Fangs' study [26], SUMO and WordNet (described in Section 3.1.2) were also examined. These studies were exploratory.…”

“…In this thesis, we aim to develop an evaluation method with a more rigorous design to ensure the validity of the measure, taking into account factors that may lead to skepticism of the applicability of the method on evaluating ontologies. Some limitations found in the initial studies [28,27,26] that need to be controlled are: First, the concepts selected for evaluation should not be too abstract since the technique was used on studies with more concrete concepts built around the basic level (the concept of basic level will be explained in Section 3.2.4). BWW, for example, might therefore not be a suitable choice of ontology for the purpose of this study if sentence verification task is selected as an evaluation technique.…”

“…Hence, we use techniques from cognitive psychology for the evaluation of ontologies, described in the following sections. An initial development of such an evaluation approach was presented in [28,27,26], and the sentence verification technique, which will be described in the latter sections, was used in these studies.…”

“…Psychological research on knowledge representation, which refers to the study of cognition and the use and access of pre-existing knowledge structures, is thus motivated to be seen as the most appropriate reference discipline for developing ontology evaluation methodologies. An initial development of such an evaluation approach is presented in Fang and Evermanns' work [28,27,26].…”