A set theory based similarity measure for text clustering and classification

Abstract

Similarity measures have long been utilized in information retrieval and machine learning domains for multi-purposes including text retrieval, text clustering, text summarization, plagiarism detection, and several other text-processing applications. However, the problem with these measures is that, until recently, there has never been one single measure recorded to be highly effective and efficient at the same time. Thus, the quest for an efficient and effective similarity measure is still an open-ended challenge. This study, in consequence, introduces a new highly-effective and time-efficient similarity measure for text clustering and classification. Furthermore, the study aims to provide a comprehensive scrutinization for seven of the most widely used similarity measures, mainly concerning their effectiveness and efficiency. Using the K-nearest neighbor algorithm (KNN) for classification, the K-means algorithm for clustering, and the bag of word (BoW) model for feature selection, all similarity measures are carefully examined in detail. The experimental evaluation has been made on two of the most popular datasets, namely, Reuters-21 and Web-KB. The obtained results confirm that the proposed set theory-based similarity measure (STB-SM), as a pre-eminent measure, outweighs all state-of-art measures significantly with regards to both effectiveness and efficiency.

Boolean logic algebra driven similarity measure for text based applications

“…Several machine learning techniques have demonstrated a surpassing performance, in the NLP field, to handle the voluminous constantly-piling data and information on the internet. Among these techniques are clustering and classification which are still commonly used in almost all scientific fields, including text mining, information retrieval, web search, pattern recognition, and biomedical based text mining ( Amer & Abdalla, 2020 ; Rachkovskij, 2017 ; Gweon, Schonlau & Steiner, 2019b ; Kanungo et al, 2002 ; Holzinger et al, 2014 ). For example, in Holzinger et al (2014) , a detailed survey in biomedical-based text mining and classification was done, while stressing the importance of involving and improving similarity measures for classification tasks.…”

“…Generally speaking, in information retrieval, the documents are drawn as vectors in the vector space model (VSM) ( Amer & Abdalla, 2020 ). In each document’s vector, each cell refers to the value of the relative feature that corresponds to the term presence/absence.…”

Ontology Matching: State of the Art, Future Challenges, and Thinking Based on Utilized Information

“…where l and l´ denote the labels of entities e and e, W denotes WordNet, syn(l) and ant(l) denote the set of synonyms and antonyms of l, Lin(l, l) denotes the information theory-based text similarity proposed by [48], and tok(l) denotes the set of words corresponding to the entity label. For example, the set of words corresponding to 'bookTitle' is {{book}, {title}}.…”

Learner question’s correctness assessment and a guided correction method: enhancing the user experience in an interactive online learning system

“…Cosine and Jaccard similarity techniques are the two text-based similarity approach which has been widely incorporated for finding similar text ( Sohangir & Wang, 2017 ; Amer & Abdalla, 2020 ). But these approaches, when applied to question-based corpus for identifying similar question text, lead to the recommendation issues, as discussed in the following subsections.…”