Bayesian Evaluation and Comparison of Ontology Alignment Systems

Abstract

Ontology alignment systems are evaluated by various performance scores, which are usually computed by a ratio related directly to the frequency of the true positives. However, such ratios provide little information regarding the uncertainty of the overall performance of the corresponding systems. The comparison is also drawn merely by the juxtaposition of computed scores, and specify that one system is superior to one another provided that its score is higher. Nonetheless, the comparison based solely on two figures would not quantify the significance of difference and would not determine the extent to which one system is better. The problem compounds for comparison over multiple benchmarks since averages and micro-averages of performance scores are considered. In this paper, the evaluation of alignment systems is translated into a statistical inference problem by introducing the notion of risk for alignment systems. The risk with respect to a performance score is shown to follow a binomial distribution and is equivalent to the complement of the score, e.g., precision risk = 1 − precision. It is also demonstrated that the maximum likelihood estimation (MLE) is precisely equivalent to the conventional evaluation by using ratios. Instead of using the MLE, the Bayesian model is developed to estimate the risk with respect to a score (or equivalently, the score itself) as a probability distribution from the performance of the systems over single or multiple benchmarks. As a result, the evaluation outcome is a distribution instead of a figure, which provides a broader view of the overall system performance. A Bayesian test is also devised to compare various systems based on their estimated risks, which can compute the confidence that one system is superior to one another. We report the result of applying the proposed methodology to multiple tracks from the ontology alignment evaluation initiative (OAEI).

Evaluating and Comparing Ontology Alignment Systems: An Mcdm Approach

“…A possible way for future research is to use statistical methods such as Bayesian models proposed in [19] that provides more information about the performance of alignment systems. There are few studies in MCDM that can handle distributional inputs.…”

“…For comparing ontology alignment systems, the dominant strategy is to first select a performance metric and then compare the systems based on the averages or micro-averages of the same metrics over multiple benchmarks. Since averaging is not statistically safe and appropriate, several statistical methods have been recently put forward for evaluation and comparison of alignment systems [17][18][19]. In particular, the Bayesian model in [19] estimates the performance of an alignment system by a distribution instead of a ratio.…”

“…Since averaging is not statistically safe and appropriate, several statistical methods have been recently put forward for evaluation and comparison of alignment systems [17][18][19]. In particular, the Bayesian model in [19] estimates the performance of an alignment system by a distribution instead of a ratio. For instance, the model outputs a precision distribution rather than a precision score.…”

A Demand-Response System for Sustainable Manufacturing Using Linked Data and Machine Learning

“…Because of the extensive applications of ontology alignment, it has been widely studied in many research works (Mohammadi 2019;Mohammadi et al 2018a,b;Mohammadi and Rezaei 2020;Zhou et al 2018). In these works, the authors also consider a considerable number of alignment systems.…”