Effects of information and machine learning algorithms on word sense disambiguation with small datasets

Abstract

Summary Current approaches to word sense disambiguation use (and often combine) various machine learning techniques. Most refer to characteristics of the ambiguity and its surrounding words and are based on thousands of examples. Unfortunately, developing large training sets is burdensome, and in response to this challenge, we investigate the use of symbolic knowledge for small datasets. A naïve Bayes classifier was trained for 15 words with 100 examples for each. Unified Medical Language System (UMLS) semantic types assigned to concepts found in the sentence and relationships between these semantic types form the knowledge base. The most frequent sense of a word served as the baseline. The effect of increasingly accurate symbolic knowledge was evaluated in nine experimental conditions. Performance was measured by accuracy based on 10-fold cross-validation. The best condition used only the semantic types of the words in the sentence. Accuracy was then on average 10% higher than the baseline; however, it varied from 8% deterioration to 29% improvement. To investigate this large variance, we performed several followup evaluations, testing additional algorithms (decision tree and neural network), and gold standards (per expert), but the results did not significantly differ. However, we noted a trend that the best disambiguation was found for words that were the least troublesome to the human evaluators. We conclude that neither algorithm nor individual human behavior cause these large differences, but that the structure of the UMLS Metathesaurus (used to represent senses of ambiguous words) contributes to inaccuracies in the gold standard, leading to varied performance of word sense disambiguation techniques.

Knowledge-based biomedical word sense disambiguation: comparison of approaches

“…This has already been shown in the literature [14-16]. The MFS indicates that usually one sense of the ambiguous word is highly represented compared to the rest of the senses.…”

“…Among the knowledge-based methods we find the Journal Descriptor Indexing method [12] and several based on graph algorithms [13]. Machine learning algorithms have been explored in several studies where alternative combinations of features are compared [14-16]; these studies obtain a performance of over 0.86 in terms of accuracy using the collection prepared by Weeber et al [17]. …”

Automated Extraction of Diagnostic Criteria From Electronic Health Records for Autism Spectrum Disorders: Development, Evaluation, and Application (Preprint)

“…We noticed such inconsistencies in prior work by us [30] and by others [19]. In the present testbed, we encountered a few omissions i.e., annotations that were identified by our parser but not by the human annotators.…”

Coordinate relationship extraction on sentence level in Chinese corpus

“…In order to compare our method with other methods in verifying the efficiency of extracting coordinate relationship, we adopt the following two kinds of methods, Naïve Bayesian (NB) [20][21][22][23] and Support Vector Machine (SVM) [24][25][26][27]. The detailed descriptions for these two methods are as follows.…”